2023-12-08T20:11:49.630 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-08T20:11:49.631 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7485056 --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} -- /home/teuthworker/archive/yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7485056/orig.config.yaml 2023-12-08T20:11:49.681 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-08T20:11:49.777 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7485056 branch: wip-yuri5-testing-2023-12-07-0740-pacific description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: null first_in_suite: false job_id: '7485056' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2023-12-07-0740-pacific ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2023-12-07-0740-pacific sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 8057 sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri5-testing-2023-12-07-0740-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-08_20:02:17 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-08T20:11:49.778 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa; will attempt to use it 2023-12-08T20:11:49.778 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks 2023-12-08T20:11:49.779 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-08T20:11:49.780 INFO:teuthology.task.internal:Checking packages... 2023-12-08T20:11:49.805 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '451ceae11b5d4caee05e94372fe658621eefc0b1' 2023-12-08T20:11:49.805 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-08T20:11:49.805 INFO:teuthology.packaging:ref: None 2023-12-08T20:11:49.805 INFO:teuthology.packaging:tag: None 2023-12-08T20:11:49.805 INFO:teuthology.packaging:branch: wip-yuri5-testing-2023-12-07-0740-pacific 2023-12-08T20:11:49.805 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T20:11:49.806 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2023-12-07-0740-pacific 2023-12-08T20:11:49.973 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-365-g451ceae1-1focal 2023-12-08T20:11:49.973 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-08T20:11:49.979 INFO:teuthology.task.internal:no buildpackages task found 2023-12-08T20:11:49.979 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-08T20:11:49.998 INFO:teuthology.task.internal:Saving configuration 2023-12-08T20:11:50.021 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-08T20:11:50.028 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-08T20:11:50.049 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7485056', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-08 20:05:20.319528', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-08T20:11:50.070 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7485056', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-08 20:05:20.321270', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-08T20:11:50.070 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-08T20:11:50.077 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-08T20:11:50.077 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-08T20:11:50.077 INFO:teuthology.run_tasks:Running task console_log... 2023-12-08T20:11:50.158 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8ccbbf20d0>, signals=[15]) 2023-12-08T20:11:50.159 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-08T20:11:50.167 INFO:teuthology.task.internal:Opening connections... 2023-12-08T20:11:50.167 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-12-08T20:11:50.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:11:50.219 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2023-12-08T20:11:50.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:11:50.273 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-08T20:11:50.308 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2023-12-08T20:11:50.323 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2023-12-08T20:11:50.323 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2023-12-08T20:11:50.381 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Ubuntu" 2023-12-08T20:11:50.381 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-08T20:11:50.381 INFO:teuthology.orchestra.run.smithi069.stdout:ID=ubuntu 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE=debian 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="20.04" 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_CODENAME=focal 2023-12-08T20:11:50.382 INFO:teuthology.orchestra.run.smithi069.stdout:UBUNTU_CODENAME=focal 2023-12-08T20:11:50.387 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2023-12-08T20:11:50.416 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2023-12-08T20:11:50.440 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2023-12-08T20:11:50.440 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2023-12-08T20:11:50.487 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="Ubuntu" 2023-12-08T20:11:50.487 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-08T20:11:50.487 INFO:teuthology.orchestra.run.smithi158.stdout:ID=ubuntu 2023-12-08T20:11:50.487 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE=debian 2023-12-08T20:11:50.487 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-08T20:11:50.487 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="20.04" 2023-12-08T20:11:50.488 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-08T20:11:50.488 INFO:teuthology.orchestra.run.smithi158.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-08T20:11:50.488 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-08T20:11:50.488 INFO:teuthology.orchestra.run.smithi158.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-08T20:11:50.488 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_CODENAME=focal 2023-12-08T20:11:50.488 INFO:teuthology.orchestra.run.smithi158.stdout:UBUNTU_CODENAME=focal 2023-12-08T20:11:50.488 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2023-12-08T20:11:50.507 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-08T20:11:50.543 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-08T20:11:50.555 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-08T20:11:50.555 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2023-12-08T20:11:50.558 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2023-12-08T20:11:50.564 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-08T20:11:50.596 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-08T20:11:50.596 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2023-12-08T20:11:50.615 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2023-12-08T20:11:50.642 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-08T20:11:50.731 INFO:teuthology.run_tasks:Running task kernel... 2023-12-08T20:11:50.834 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-08T20:11:50.835 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-08T20:11:50.835 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-08T20:11:50.836 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-08T20:11:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:11:50.841 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-12-08T20:11:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:11:50.843 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2023-12-08T20:11:50.849 INFO:teuthology.orchestra.run.smithi069.stdout:5.4.0-124-generic 2023-12-08T20:11:50.849 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.4.0-124-generic 2023-12-08T20:11:50.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2023-12-08T20:11:50.891 INFO:teuthology.orchestra.run.smithi158.stdout:5.4.0-124-generic 2023-12-08T20:11:50.892 INFO:teuthology.task.kernel:Running kernel on smithi158: 5.4.0-124-generic 2023-12-08T20:11:50.892 DEBUG:teuthology.orchestra.run.smithi158:> sudo apt-get clean 2023-12-08T20:11:50.957 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-12-08T20:11:51.001 DEBUG:teuthology.orchestra.run.smithi158:> sudo apt-get update 2023-12-08T20:11:51.155 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-08T20:11:51.155 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-12-08T20:11:51.155 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-12-08T20:11:51.178 INFO:teuthology.orchestra.run.smithi158.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-12-08T20:11:51.215 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-12-08T20:11:51.294 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-08T20:11:51.378 INFO:teuthology.orchestra.run.smithi158.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-12-08T20:11:51.506 INFO:teuthology.orchestra.run.smithi158.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,610 kB] 2023-12-08T20:11:51.531 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [686 kB] 2023-12-08T20:11:51.619 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,610 kB] 2023-12-08T20:11:51.661 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [402 kB] 2023-12-08T20:11:51.667 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-12-08T20:11:51.667 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,411 kB] 2023-12-08T20:11:51.670 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [917 kB] 2023-12-08T20:11:51.699 INFO:teuthology.orchestra.run.smithi158.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [686 kB] 2023-12-08T20:11:51.701 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-12-08T20:11:51.701 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [336 kB] 2023-12-08T20:11:51.703 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-12-08T20:11:51.704 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [915 kB] 2023-12-08T20:11:51.708 INFO:teuthology.orchestra.run.smithi158.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [402 kB] 2023-12-08T20:11:51.709 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [634 kB] 2023-12-08T20:11:51.715 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [192 kB] 2023-12-08T20:11:51.715 INFO:teuthology.orchestra.run.smithi158.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-12-08T20:11:51.715 INFO:teuthology.orchestra.run.smithi158.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,411 kB] 2023-12-08T20:11:51.716 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-12-08T20:11:51.717 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-12-08T20:11:51.717 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-12-08T20:11:51.728 INFO:teuthology.orchestra.run.smithi158.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-12-08T20:11:51.738 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,015 kB] 2023-12-08T20:11:51.764 INFO:teuthology.orchestra.run.smithi158.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-12-08T20:11:51.764 INFO:teuthology.orchestra.run.smithi158.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [336 kB] 2023-12-08T20:11:51.771 INFO:teuthology.orchestra.run.smithi158.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-12-08T20:11:51.771 INFO:teuthology.orchestra.run.smithi158.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [634 kB] 2023-12-08T20:11:51.777 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [487 kB] 2023-12-08T20:11:51.777 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-12-08T20:11:51.778 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-12-08T20:11:51.780 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-12-08T20:11:51.780 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,562 kB] 2023-12-08T20:11:51.790 INFO:teuthology.orchestra.run.smithi158.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [915 kB] 2023-12-08T20:11:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-12-08T20:11:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [358 kB] 2023-12-08T20:11:51.800 INFO:teuthology.orchestra.run.smithi158.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [192 kB] 2023-12-08T20:11:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-12-08T20:11:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,140 kB] 2023-12-08T20:11:51.803 INFO:teuthology.orchestra.run.smithi158.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-12-08T20:11:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [762 kB] 2023-12-08T20:11:51.807 INFO:teuthology.orchestra.run.smithi158.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-12-08T20:11:51.807 INFO:teuthology.orchestra.run.smithi158.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-12-08T20:11:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [273 kB] 2023-12-08T20:11:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-12-08T20:11:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-12-08T20:11:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-12-08T20:11:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-12-08T20:11:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-12-08T20:11:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-12-08T20:11:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-12-08T20:11:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-12-08T20:11:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-12-08T20:11:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-12-08T20:11:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-12-08T20:11:51.848 INFO:teuthology.orchestra.run.smithi158.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-12-08T20:11:51.848 INFO:teuthology.orchestra.run.smithi158.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-12-08T20:11:51.874 INFO:teuthology.orchestra.run.smithi158.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,015 kB] 2023-12-08T20:11:52.259 INFO:teuthology.orchestra.run.smithi158.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [917 kB] 2023-12-08T20:11:52.288 INFO:teuthology.orchestra.run.smithi158.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [487 kB] 2023-12-08T20:11:52.301 INFO:teuthology.orchestra.run.smithi158.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-12-08T20:11:52.302 INFO:teuthology.orchestra.run.smithi158.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,562 kB] 2023-12-08T20:11:52.367 INFO:teuthology.orchestra.run.smithi158.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-12-08T20:11:52.369 INFO:teuthology.orchestra.run.smithi158.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [358 kB] 2023-12-08T20:11:52.389 INFO:teuthology.orchestra.run.smithi158.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-12-08T20:11:52.554 INFO:teuthology.orchestra.run.smithi158.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,140 kB] 2023-12-08T20:11:53.190 INFO:teuthology.orchestra.run.smithi158.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [762 kB] 2023-12-08T20:11:53.229 INFO:teuthology.orchestra.run.smithi158.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [273 kB] 2023-12-08T20:11:53.242 INFO:teuthology.orchestra.run.smithi158.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-12-08T20:11:53.243 INFO:teuthology.orchestra.run.smithi158.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-12-08T20:11:53.244 INFO:teuthology.orchestra.run.smithi158.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-12-08T20:11:53.245 INFO:teuthology.orchestra.run.smithi158.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-12-08T20:11:53.245 INFO:teuthology.orchestra.run.smithi158.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-12-08T20:11:53.246 INFO:teuthology.orchestra.run.smithi158.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-12-08T20:11:53.272 INFO:teuthology.orchestra.run.smithi158.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-12-08T20:11:53.273 INFO:teuthology.orchestra.run.smithi158.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-12-08T20:11:53.438 INFO:teuthology.orchestra.run.smithi158.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-12-08T20:11:53.600 INFO:teuthology.orchestra.run.smithi158.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-12-08T20:11:53.650 INFO:teuthology.orchestra.run.smithi158.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-12-08T20:11:55.562 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 18.4 MB in 3s (7,056 kB/s) 2023-12-08T20:11:55.843 INFO:teuthology.orchestra.run.smithi158.stdout:Fetched 18.4 MB in 3s (6,920 kB/s) 2023-12-08T20:11:56.955 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-12-08T20:11:56.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-08T20:11:57.049 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-12-08T20:11:57.254 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-12-08T20:11:57.254 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-12-08T20:11:57.348 INFO:teuthology.orchestra.run.smithi158.stdout:Reading package lists... 2023-12-08T20:11:57.375 DEBUG:teuthology.orchestra.run.smithi158:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-08T20:11:57.433 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2023-12-08T20:11:57.433 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-08T20:11:57.433 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-generic linux-image-5.4.0-169-generic 2023-12-08T20:11:57.433 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-5.4.0-169-generic linux-modules-extra-5.4.0-169-generic 2023-12-08T20:11:57.434 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2023-12-08T20:11:57.434 INFO:teuthology.orchestra.run.smithi069.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-12-08T20:11:57.434 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2023-12-08T20:11:57.434 INFO:teuthology.orchestra.run.smithi069.stdout: thermald 2023-12-08T20:11:57.459 INFO:teuthology.orchestra.run.smithi158.stdout:Reading package lists... 2023-12-08T20:11:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2023-12-08T20:11:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-08T20:11:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.4.0-169-generic linux-modules-5.4.0-169-generic 2023-12-08T20:11:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-extra-5.4.0-169-generic 2023-12-08T20:11:57.481 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages will be upgraded: 2023-12-08T20:11:57.481 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-08T20:11:57.659 INFO:teuthology.orchestra.run.smithi158.stdout:Building dependency tree... 2023-12-08T20:11:57.660 INFO:teuthology.orchestra.run.smithi158.stdout:Reading state information... 2023-12-08T20:11:57.665 INFO:teuthology.orchestra.run.smithi069.stdout:3 upgraded, 5 newly installed, 0 to remove and 280 not upgraded. 2023-12-08T20:11:57.666 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 77.1 MB of archives. 2023-12-08T20:11:57.666 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 380 MB of additional disk space will be used. 2023-12-08T20:11:57.666 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-169-generic amd64 5.4.0-169.187 [15.0 MB] 2023-12-08T20:11:57.837 INFO:teuthology.orchestra.run.smithi158.stdout:The following additional packages will be installed: 2023-12-08T20:11:57.837 INFO:teuthology.orchestra.run.smithi158.stdout: linux-generic linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-08T20:11:57.837 INFO:teuthology.orchestra.run.smithi158.stdout: linux-headers-generic linux-image-5.4.0-169-generic 2023-12-08T20:11:57.837 INFO:teuthology.orchestra.run.smithi158.stdout: linux-modules-5.4.0-169-generic linux-modules-extra-5.4.0-169-generic 2023-12-08T20:11:57.838 INFO:teuthology.orchestra.run.smithi158.stdout:Suggested packages: 2023-12-08T20:11:57.838 INFO:teuthology.orchestra.run.smithi158.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-12-08T20:11:57.838 INFO:teuthology.orchestra.run.smithi158.stdout:Recommended packages: 2023-12-08T20:11:57.839 INFO:teuthology.orchestra.run.smithi158.stdout: thermald 2023-12-08T20:11:57.906 INFO:teuthology.orchestra.run.smithi158.stdout:The following NEW packages will be installed: 2023-12-08T20:11:57.906 INFO:teuthology.orchestra.run.smithi158.stdout: linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-08T20:11:57.906 INFO:teuthology.orchestra.run.smithi158.stdout: linux-image-5.4.0-169-generic linux-modules-5.4.0-169-generic 2023-12-08T20:11:57.906 INFO:teuthology.orchestra.run.smithi158.stdout: linux-modules-extra-5.4.0-169-generic 2023-12-08T20:11:57.907 INFO:teuthology.orchestra.run.smithi158.stdout:The following packages will be upgraded: 2023-12-08T20:11:57.908 INFO:teuthology.orchestra.run.smithi158.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-08T20:11:57.959 INFO:teuthology.orchestra.run.smithi158.stdout:3 upgraded, 5 newly installed, 0 to remove and 280 not upgraded. 2023-12-08T20:11:57.959 INFO:teuthology.orchestra.run.smithi158.stdout:Need to get 77.1 MB of archives. 2023-12-08T20:11:57.959 INFO:teuthology.orchestra.run.smithi158.stdout:After this operation, 380 MB of additional disk space will be used. 2023-12-08T20:11:57.959 INFO:teuthology.orchestra.run.smithi158.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-169-generic amd64 5.4.0-169.187 [15.0 MB] 2023-12-08T20:11:58.281 INFO:teuthology.orchestra.run.smithi158.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-169-generic amd64 5.4.0-169.187 [10.5 MB] 2023-12-08T20:11:58.372 INFO:teuthology.orchestra.run.smithi158.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-169-generic amd64 5.4.0-169.187 [39.2 MB] 2023-12-08T20:11:58.798 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-169-generic amd64 5.4.0-169.187 [10.5 MB] 2023-12-08T20:11:58.910 INFO:teuthology.orchestra.run.smithi158.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.169.167 [1,900 B] 2023-12-08T20:11:58.910 INFO:teuthology.orchestra.run.smithi158.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.169.167 [2,520 B] 2023-12-08T20:11:58.911 INFO:teuthology.orchestra.run.smithi158.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169 all 5.4.0-169.187 [11.0 MB] 2023-12-08T20:11:59.006 INFO:teuthology.orchestra.run.smithi158.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169-generic amd64 5.4.0-169.187 [1,374 kB] 2023-12-08T20:11:59.016 INFO:teuthology.orchestra.run.smithi158.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.169.167 [2,392 B] 2023-12-08T20:11:59.069 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-169-generic amd64 5.4.0-169.187 [39.2 MB] 2023-12-08T20:11:59.730 INFO:teuthology.orchestra.run.smithi158.stdout:Fetched 77.1 MB in 1s (70.0 MB/s) 2023-12-08T20:11:59.940 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package linux-modules-5.4.0-169-generic. 2023-12-08T20:12:00.360 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.169.167 [1,900 B] 2023-12-08T20:12:00.362 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.169.167 [2,520 B] 2023-12-08T20:12:00.362 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169 all 5.4.0-169.187 [11.0 MB] 2023-12-08T20:12:00.654 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169-generic amd64 5.4.0-169.187 [1,374 kB] 2023-12-08T20:12:00.696 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.169.167 [2,392 B] 2023-12-08T20:12:01.349 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 77.1 MB in 3s (24.0 MB/s) 2023-12-08T20:12:01.536 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-5.4.0-169-generic. 2023-12-08T20:12:01.660 INFO:teuthology.orchestra.run.smithi158.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-12-08T20:12:01.665 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../0-linux-modules-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:01.746 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:03.259 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-12-08T20:12:03.264 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../0-linux-modules-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:03.341 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:03.846 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package linux-image-5.4.0-169-generic. 2023-12-08T20:12:03.861 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../1-linux-image-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:03.968 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:05.074 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-169-generic. 2023-12-08T20:12:05.089 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:05.132 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:05.193 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-image-5.4.0-169-generic. 2023-12-08T20:12:05.210 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../1-linux-image-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:05.313 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:06.371 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-169-generic. 2023-12-08T20:12:06.386 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:06.430 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:09.969 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../3-linux-generic_5.4.0.169.167_amd64.deb ... 2023-12-08T20:12:10.096 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-08T20:12:10.430 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.169.167_amd64.deb ... 2023-12-08T20:12:10.557 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-image-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-08T20:12:10.902 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package linux-headers-5.4.0-169. 2023-12-08T20:12:10.918 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../5-linux-headers-5.4.0-169_5.4.0-169.187_all.deb ... 2023-12-08T20:12:10.960 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-08T20:12:11.130 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../3-linux-generic_5.4.0.169.167_amd64.deb ... 2023-12-08T20:12:11.249 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-08T20:12:11.574 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.169.167_amd64.deb ... 2023-12-08T20:12:11.702 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-08T20:12:12.047 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.4.0-169. 2023-12-08T20:12:12.060 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../5-linux-headers-5.4.0-169_5.4.0-169.187_all.deb ... 2023-12-08T20:12:12.097 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-08T20:12:16.349 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package linux-headers-5.4.0-169-generic. 2023-12-08T20:12:16.370 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../6-linux-headers-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:16.407 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:17.531 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.4.0-169-generic. 2023-12-08T20:12:17.553 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../6-linux-headers-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-08T20:12:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:19.120 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.169.167_amd64.deb ... 2023-12-08T20:12:19.241 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking linux-headers-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-08T20:12:19.563 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-08T20:12:19.687 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:19.813 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:20.338 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.169.167_amd64.deb ... 2023-12-08T20:12:20.443 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-08T20:12:20.762 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-08T20:12:20.914 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:21.040 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:21.082 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:22.212 INFO:teuthology.orchestra.run.smithi158.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-169-generic 2023-12-08T20:12:22.212 INFO:teuthology.orchestra.run.smithi158.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:22.275 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:22.342 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-headers-generic (5.4.0.169.167) ... 2023-12-08T20:12:22.493 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:23.670 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-image-generic (5.4.0.169.167) ... 2023-12-08T20:12:23.708 INFO:teuthology.orchestra.run.smithi069.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-169-generic 2023-12-08T20:12:23.708 INFO:teuthology.orchestra.run.smithi069.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:23.796 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up linux-generic (5.4.0.169.167) ... 2023-12-08T20:12:24.422 INFO:teuthology.orchestra.run.smithi158.stdout:Processing triggers for linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:24.520 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-generic (5.4.0.169.167) ... 2023-12-08T20:12:24.646 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:25.016 INFO:teuthology.orchestra.run.smithi158.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-08T20:12:25.016 INFO:teuthology.orchestra.run.smithi158.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:25.846 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-generic (5.4.0.169.167) ... 2023-12-08T20:12:25.964 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-generic (5.4.0.169.167) ... 2023-12-08T20:12:26.091 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-08T20:12:26.151 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-08T20:12:26.151 INFO:teuthology.orchestra.run.smithi069.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:47.254 INFO:teuthology.orchestra.run.smithi158.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-08T20:12:47.254 INFO:teuthology.orchestra.run.smithi158.stdout:Sourcing file `/etc/default/grub' 2023-12-08T20:12:47.262 INFO:teuthology.orchestra.run.smithi158.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-08T20:12:47.282 INFO:teuthology.orchestra.run.smithi158.stdout:Generating grub configuration file ... 2023-12-08T20:12:47.609 INFO:teuthology.orchestra.run.smithi158.stdout:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-08T20:12:47.642 INFO:teuthology.orchestra.run.smithi158.stdout:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:48.029 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-08T20:12:48.029 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub' 2023-12-08T20:12:48.035 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-08T20:12:48.043 INFO:teuthology.orchestra.run.smithi158.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-08T20:12:48.045 INFO:teuthology.orchestra.run.smithi069.stdout:Generating grub configuration file ... 2023-12-08T20:12:48.055 INFO:teuthology.orchestra.run.smithi158.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-08T20:12:48.364 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-08T20:12:48.389 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:48.768 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-08T20:12:48.780 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-08T20:12:49.839 INFO:teuthology.orchestra.run.smithi158.stdout:done 2023-12-08T20:12:50.287 DEBUG:teuthology.orchestra.run.smithi158:> dpkg -s linux-image-generic 2023-12-08T20:12:50.345 INFO:teuthology.orchestra.run.smithi158.stdout:Package: linux-image-generic 2023-12-08T20:12:50.345 INFO:teuthology.orchestra.run.smithi158.stdout:Status: install ok installed 2023-12-08T20:12:50.345 INFO:teuthology.orchestra.run.smithi158.stdout:Priority: optional 2023-12-08T20:12:50.345 INFO:teuthology.orchestra.run.smithi158.stdout:Section: kernel 2023-12-08T20:12:50.345 INFO:teuthology.orchestra.run.smithi158.stdout:Installed-Size: 20 2023-12-08T20:12:50.345 INFO:teuthology.orchestra.run.smithi158.stdout:Maintainer: Ubuntu Kernel Team 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: amd64 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:Source: linux-meta 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:Version: 5.4.0.169.167 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:Recommends: thermald 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:Description: Generic Linux kernel image 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout: This package will always depend on the latest generic kernel image 2023-12-08T20:12:50.346 INFO:teuthology.orchestra.run.smithi158.stdout: available. 2023-12-08T20:12:50.346 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-169-generic 2023-12-08T20:12:50.347 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-08T20:12:50.347 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-08T20:12:50.347 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-08T20:12:50.347 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi158.front.sepia.ceph.com, path=None, version=distro) 2023-12-08T20:12:50.347 DEBUG:teuthology.orchestra.run.smithi158:> sudo apt-get clean 2023-12-08T20:12:50.451 DEBUG:teuthology.orchestra.run.smithi158:> sudo apt-get update 2023-12-08T20:12:50.713 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-08T20:12:50.713 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-08T20:12:50.783 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-08T20:12:50.864 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-08T20:12:51.091 INFO:teuthology.orchestra.run.smithi069.stdout:done 2023-12-08T20:12:51.479 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2023-12-08T20:12:51.531 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 20 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.4.0.169.167 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2023-12-08T20:12:51.532 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2023-12-08T20:12:51.533 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-169-generic 2023-12-08T20:12:51.533 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-08T20:12:51.533 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-08T20:12:51.533 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-08T20:12:51.533 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2023-12-08T20:12:51.533 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2023-12-08T20:12:51.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-12-08T20:12:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-08T20:12:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-08T20:12:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-08T20:12:52.025 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-08T20:12:52.484 INFO:teuthology.orchestra.run.smithi158.stdout:Reading package lists... 2023-12-08T20:12:52.509 DEBUG:teuthology.orchestra.run.smithi158:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-08T20:12:52.576 INFO:teuthology.orchestra.run.smithi158.stdout:Reading package lists... 2023-12-08T20:12:52.781 INFO:teuthology.orchestra.run.smithi158.stdout:Building dependency tree... 2023-12-08T20:12:52.781 INFO:teuthology.orchestra.run.smithi158.stdout:Reading state information... 2023-12-08T20:12:53.004 INFO:teuthology.orchestra.run.smithi158.stdout:linux-image-generic is already the newest version (5.4.0.169.167). 2023-12-08T20:12:53.004 INFO:teuthology.orchestra.run.smithi158.stdout:0 upgraded, 0 newly installed, 0 to remove and 280 not upgraded. 2023-12-08T20:12:53.006 DEBUG:teuthology.orchestra.run.smithi158:> dpkg -s linux-image-generic 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Package: linux-image-generic 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Status: install ok installed 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Priority: optional 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Section: kernel 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Installed-Size: 20 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Maintainer: Ubuntu Kernel Team 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: amd64 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Source: linux-meta 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Version: 5.4.0.169.167 2023-12-08T20:12:53.026 INFO:teuthology.orchestra.run.smithi158.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-08T20:12:53.027 INFO:teuthology.orchestra.run.smithi158.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-08T20:12:53.027 INFO:teuthology.orchestra.run.smithi158.stdout:Recommends: thermald 2023-12-08T20:12:53.027 INFO:teuthology.orchestra.run.smithi158.stdout:Description: Generic Linux kernel image 2023-12-08T20:12:53.027 INFO:teuthology.orchestra.run.smithi158.stdout: This package will always depend on the latest generic kernel image 2023-12-08T20:12:53.027 INFO:teuthology.orchestra.run.smithi158.stdout: available. 2023-12-08T20:12:53.027 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2023-12-08T20:12:53.032 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.Zy0LzHR1t0 2023-12-08T20:12:53.032 DEBUG:teuthology.orchestra.run.smithi158:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Zy0LzHR1t0 2023-12-08T20:12:53.098 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.Zy0LzHR1t0 2023-12-08T20:12:53.136 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.Zy0LzHR1t0 is 9KB 2023-12-08T20:12:53.150 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.Zy0LzHR1t0 2023-12-08T20:12:53.156 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-08T20:12:53.213 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:12:53.213 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-08T20:12:53.213 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-08T20:12:53.257 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-169-generic 2023-12-08T20:12:53.257 DEBUG:teuthology.orchestra.run.smithi158:> sudo update-grub 2023-12-08T20:12:53.633 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-12-08T20:12:53.658 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-08T20:12:53.726 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-12-08T20:12:53.930 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-12-08T20:12:53.931 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-12-08T20:12:54.157 INFO:teuthology.orchestra.run.smithi069.stdout:linux-image-generic is already the newest version (5.4.0.169.167). 2023-12-08T20:12:54.157 INFO:teuthology.orchestra.run.smithi069.stdout:0 upgraded, 0 newly installed, 0 to remove and 280 not upgraded. 2023-12-08T20:12:54.158 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2023-12-08T20:12:54.177 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2023-12-08T20:12:54.177 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 20 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.4.0.169.167 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2023-12-08T20:12:54.178 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2023-12-08T20:12:54.179 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-12-08T20:12:54.183 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.NXWTxdMuJY 2023-12-08T20:12:54.183 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NXWTxdMuJY 2023-12-08T20:12:54.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.NXWTxdMuJY 2023-12-08T20:12:54.279 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.NXWTxdMuJY is 9KB 2023-12-08T20:12:54.294 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.NXWTxdMuJY 2023-12-08T20:12:54.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-08T20:12:54.358 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:12:54.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-08T20:12:54.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-08T20:12:54.384 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-169-generic 2023-12-08T20:12:54.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo update-grub 2023-12-08T20:12:54.563 INFO:teuthology.orchestra.run.smithi158.stderr:Sourcing file `/etc/default/grub' 2023-12-08T20:12:54.569 INFO:teuthology.orchestra.run.smithi158.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-08T20:12:54.579 INFO:teuthology.orchestra.run.smithi158.stderr:Generating grub configuration file ... 2023-12-08T20:12:54.819 INFO:teuthology.orchestra.run.smithi158.stderr:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-08T20:12:54.829 INFO:teuthology.orchestra.run.smithi158.stderr:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:55.218 INFO:teuthology.orchestra.run.smithi158.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-08T20:12:55.230 INFO:teuthology.orchestra.run.smithi158.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-08T20:12:55.638 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub' 2023-12-08T20:12:55.644 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-08T20:12:55.655 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2023-12-08T20:12:55.900 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-08T20:12:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-08T20:12:55.981 INFO:teuthology.orchestra.run.smithi158.stderr:done 2023-12-08T20:12:55.982 DEBUG:teuthology.orchestra.run.smithi158:> sudo shutdown -r now 2023-12-08T20:12:56.251 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-08T20:12:56.264 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-08T20:12:57.449 INFO:teuthology.orchestra.run.smithi069.stderr:done 2023-12-08T20:12:57.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2023-12-08T20:13:26.010 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-08T20:13:26.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-12-08T20:13:26.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:13:27.454 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-08T20:13:27.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-08T20:13:27.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:14:00.551 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-12-08T20:14:09.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-12-08T20:14:09.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:14:12.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-12-08T20:14:24.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-12-08T20:14:24.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:14:27.458 DEBUG:teuthology.orchestra.remote:timed out 2023-12-08T20:14:36.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-08T20:14:36.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:14:37.146 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-12-08T20:14:38.144 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-08T20:14:38.144 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.4.0-169-generic"... 2023-12-08T20:14:38.144 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-12-08T20:14:38.150 INFO:teuthology.orchestra.run.smithi069.stdout:5.4.0-169-generic 2023-12-08T20:14:38.150 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-169-generic vs 5.4.0-169-generic 2023-12-08T20:14:38.151 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-08T20:14:38.151 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-08T20:14:39.152 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-08T20:14:39.152 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-08T20:14:39.250 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2023-12-08T20:14:39.289 DEBUG:teuthology.parallel:result is None 2023-12-08T20:14:43.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-12-08T20:14:58.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-12-08T20:14:58.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:14:58.265 DEBUG:teuthology.orchestra.run.smithi158:> true 2023-12-08T20:14:58.837 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-12-08T20:14:58.837 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.4.0-169-generic"... 2023-12-08T20:14:58.838 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2023-12-08T20:14:58.844 INFO:teuthology.orchestra.run.smithi158.stdout:5.4.0-169-generic 2023-12-08T20:14:58.845 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-169-generic vs 5.4.0-169-generic 2023-12-08T20:14:58.845 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-08T20:14:58.845 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-08T20:14:59.846 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-08T20:14:59.846 DEBUG:teuthology.orchestra.run.smithi158:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-08T20:14:59.924 INFO:teuthology.orchestra.run.smithi158.stdout:ttyS1 2023-12-08T20:14:59.948 DEBUG:teuthology.parallel:result is None 2023-12-08T20:14:59.949 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-08T20:14:59.956 INFO:teuthology.task.internal:Creating test directory... 2023-12-08T20:14:59.957 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-08T20:14:59.961 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-08T20:14:59.968 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-08T20:14:59.974 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-08T20:14:59.985 INFO:teuthology.task.internal:Creating archive directory... 2023-12-08T20:14:59.985 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-08T20:15:00.009 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-08T20:15:00.039 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-08T20:15:00.046 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-08T20:15:00.046 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-08T20:15:00.048 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-08T20:15:00.073 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T20:15:00.084 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T20:15:00.109 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T20:15:00.121 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T20:15:00.123 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-08T20:15:00.130 INFO:teuthology.task.internal:Configuring sudo... 2023-12-08T20:15:00.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-08T20:15:00.132 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-08T20:15:00.149 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-08T20:15:00.180 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-08T20:15:00.181 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-08T20:15:00.193 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-08T20:15:00.200 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-08T20:15:00.208 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-08T20:15:00.214 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:15:00.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-08T20:15:00.281 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-08T20:15:00.291 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-08T20:15:00.341 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:15:00.341 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-08T20:15:00.366 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2023-12-08T20:15:00.368 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2023-12-08T20:15:00.416 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-08T20:15:00.425 INFO:teuthology.task.internal:Starting timer... 2023-12-08T20:15:00.425 INFO:teuthology.run_tasks:Running task pcp... 2023-12-08T20:15:00.441 INFO:teuthology.run_tasks:Running task selinux... 2023-12-08T20:15:00.450 DEBUG:teuthology.task.selinux:Excluding smithi069: OS 'ubuntu' does not support SELinux 2023-12-08T20:15:00.450 DEBUG:teuthology.task.selinux:Excluding smithi158: OS 'ubuntu' does not support SELinux 2023-12-08T20:15:00.451 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-08T20:15:00.451 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-08T20:15:00.451 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-08T20:15:00.451 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-08T20:15:00.461 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-08T20:15:00.461 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-08T20:15:00.489 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-08T20:15:00.491 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi158.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-08T20:21:05.290 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi158.front.sepia.ceph.com')] 2023-12-08T20:21:05.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-08T20:21:05.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:21:05.340 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-12-08T20:21:05.463 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-12-08T20:21:05.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-12-08T20:21:05.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T20:21:05.512 DEBUG:teuthology.orchestra.run.smithi158:> true 2023-12-08T20:21:05.636 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-12-08T20:21:05.636 INFO:teuthology.run_tasks:Running task clock... 2023-12-08T20:21:05.647 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-08T20:21:05.647 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-08T20:21:05.648 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T20:21:05.650 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-08T20:21:05.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T20:21:05.682 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-08T20:21:05.683 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Command line: ntpd -gq 2023-12-08T20:21:05.683 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: proto: precision = 0.070 usec (-24) 2023-12-08T20:21:05.683 INFO:teuthology.orchestra.run.smithi069.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-08T20:21:05.684 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-08T20:21:05.684 INFO:teuthology.orchestra.run.smithi069.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: restrict ::: KOD does nothing without LIMITED. 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Listen and drop on 0 v6wildcard [::]:123 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Listen normally on 2 lo 127.0.0.1:123 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Listen normally on 3 enp3s0f1 172.21.15.69:123 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Listen normally on 4 lo [::1]:123 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d737%5]:123 2023-12-08T20:21:05.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:05 ntpd[23625]: Listening on routing socket on fd #22 for interface updates 2023-12-08T20:21:05.686 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-08T20:21:05.686 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Command line: ntpd -gq 2023-12-08T20:21:05.686 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: proto: precision = 0.087 usec (-23) 2023-12-08T20:21:05.686 INFO:teuthology.orchestra.run.smithi158.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-08T20:21:05.687 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-08T20:21:05.687 INFO:teuthology.orchestra.run.smithi158.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: restrict ::: KOD does nothing without LIMITED. 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Listen and drop on 0 v6wildcard [::]:123 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Listen normally on 2 lo 127.0.0.1:123 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Listen normally on 3 enp3s0f1 172.21.15.158:123 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Listen normally on 4 lo [::1]:123 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:726f%5]:123 2023-12-08T20:21:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:05 ntpd[23649]: Listening on routing socket on fd #22 for interface updates 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:06.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:06 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:07.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:07 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:07.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:07 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:07.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:07 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:07.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:07 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:07.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:07 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:07.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:07 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:07.685 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:07 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:07.685 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:07 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:07.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:07 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:07.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:07 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:07.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:07 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:07.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:07 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:08.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:08.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:08.686 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:08.686 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:08.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:08 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:08.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:08 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:08.687 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:08 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:09.683 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:09 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:09.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:09 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:09.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:09 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:09.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:09 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:09.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:09 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:09.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:09 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:09.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:09.688 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:09.688 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:09.688 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:09 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:10.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:10.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:10.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:10.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:10.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:10.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:10.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:10.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:10.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:10.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:10.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:10.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:10 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:10.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:10 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:10.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:10 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:10.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:10 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:10.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:10 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:10.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:10 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:10.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:10 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:11.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:11 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:11.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:11 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:11.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:11 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:11.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:11 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:11.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:11 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:11.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:11 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:11.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:11.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:11.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:11.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:11.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:11.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:11.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:11.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:11.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:11.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:11.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:11.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:11 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:12.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:12.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:12.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:12.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:12.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:12.684 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:12.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:12.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:12.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:12.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:12.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:12.685 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: 8 Dec 20:21:12 ntpd[23625]: ntpd: time slew +0.003594 s 2023-12-08T20:21:12.686 INFO:teuthology.orchestra.run.smithi069.stdout:ntpd: time slew +0.003594s 2023-12-08T20:21:12.686 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-08T20:21:12.686 INFO:teuthology.orchestra.run.smithi069.stderr: 8 Dec 20:21:12 ntpd[23625]: can't open /var/log/ntpstats/loopstats.20231208: Permission denied 2023-12-08T20:21:12.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:12 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:12.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:12 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:12.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:12 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:12.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:12 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:12.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:12 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:12.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:12 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: remote refid st t when poll reach delay offset jitter 2023-12-08T20:21:12.756 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================== 2023-12-08T20:21:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: can't open /var/log/ntpstats/rawstats.20231208: Permission denied 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-08T20:21:13.686 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: can't open /var/log/ntpstats/peerstats.20231208: Permission denied 2023-12-08T20:21:13.687 INFO:teuthology.orchestra.run.smithi158.stdout: 8 Dec 20:21:13 ntpd[23649]: ntpd: time slew +0.002331 s 2023-12-08T20:21:13.687 INFO:teuthology.orchestra.run.smithi158.stdout:ntpd: time slew +0.002331s 2023-12-08T20:21:13.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-08T20:21:13.687 INFO:teuthology.orchestra.run.smithi158.stderr: 8 Dec 20:21:13 ntpd[23649]: can't open /var/log/ntpstats/loopstats.20231208: Permission denied 2023-12-08T20:21:13.751 INFO:teuthology.orchestra.run.smithi158.stdout: remote refid st t when poll reach delay offset jitter 2023-12-08T20:21:13.751 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================== 2023-12-08T20:21:13.751 INFO:teuthology.orchestra.run.smithi158.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:13.751 INFO:teuthology.orchestra.run.smithi158.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:13.751 INFO:teuthology.orchestra.run.smithi158.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:13.751 INFO:teuthology.orchestra.run.smithi158.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-08T20:21:13.752 INFO:teuthology.run_tasks:Running task install... 2023-12-08T20:21:13.761 DEBUG:teuthology.task.install:project ceph 2023-12-08T20:21:13.761 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}} 2023-12-08T20:21:13.762 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'} 2023-12-08T20:21:13.762 INFO:teuthology.task.install:Using flavor: default 2023-12-08T20:21:13.768 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-08T20:21:13.768 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-08T20:21:13.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-key list | grep Ceph 2023-12-08T20:21:13.769 DEBUG:teuthology.orchestra.run.smithi158:> sudo apt-key list | grep Ceph 2023-12-08T20:21:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-12-08T20:21:13.873 INFO:teuthology.orchestra.run.smithi158.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-12-08T20:21:13.925 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-08T20:21:13.925 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph.com (release key) 2023-12-08T20:21:13.926 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-08T20:21:13.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T20:21:13.936 INFO:teuthology.orchestra.run.smithi158.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-08T20:21:13.936 INFO:teuthology.orchestra.run.smithi158.stdout:uid [ unknown] Ceph.com (release key) 2023-12-08T20:21:13.936 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-08T20:21:13.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T20:21:14.085 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default/ 2023-12-08T20:21:14.086 INFO:teuthology.task.install.deb:Package version is 16.2.14-365-g451ceae1-1focal 2023-12-08T20:21:14.091 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default/ 2023-12-08T20:21:14.092 INFO:teuthology.task.install.deb:Package version is 16.2.14-365-g451ceae1-1focal 2023-12-08T20:21:14.256 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:21:14.256 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-08T20:21:14.268 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:21:14.268 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-08T20:21:14.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-12-08T20:21:14.290 DEBUG:teuthology.orchestra.run.smithi158:> sudo apt-get update 2023-12-08T20:21:14.415 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-08T20:21:14.415 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-08T20:21:14.423 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-08T20:21:14.441 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-08T20:21:14.488 INFO:teuthology.orchestra.run.smithi158.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal InRelease 2023-12-08T20:21:14.509 INFO:teuthology.orchestra.run.smithi158.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal Release [7,680 B] 2023-12-08T20:21:14.533 INFO:teuthology.orchestra.run.smithi158.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal Release.gpg 2023-12-08T20:21:14.536 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal InRelease 2023-12-08T20:21:14.560 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-08T20:21:14.560 INFO:teuthology.orchestra.run.smithi158.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main i386 Packages [3,106 B] 2023-12-08T20:21:14.572 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal Release [7,680 B] 2023-12-08T20:21:14.592 INFO:teuthology.orchestra.run.smithi158.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 Packages [17.2 kB] 2023-12-08T20:21:14.605 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-08T20:21:14.611 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal Release.gpg 2023-12-08T20:21:14.630 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-08T20:21:14.647 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main i386 Packages [3,106 B] 2023-12-08T20:21:14.696 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 Packages [17.2 kB] 2023-12-08T20:21:14.711 INFO:teuthology.orchestra.run.smithi158.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-08T20:21:15.069 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 28.0 kB in 1s (44.4 kB/s) 2023-12-08T20:21:15.211 INFO:teuthology.orchestra.run.smithi158.stdout:Fetched 28.0 kB in 1s (36.9 kB/s) 2023-12-08T20:21:16.148 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-12-08T20:21:16.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-365-g451ceae1-1focal cephadm=16.2.14-365-g451ceae1-1focal ceph-mds=16.2.14-365-g451ceae1-1focal ceph-mgr=16.2.14-365-g451ceae1-1focal ceph-common=16.2.14-365-g451ceae1-1focal ceph-fuse=16.2.14-365-g451ceae1-1focal ceph-test=16.2.14-365-g451ceae1-1focal radosgw=16.2.14-365-g451ceae1-1focal python3-rados=16.2.14-365-g451ceae1-1focal python3-rgw=16.2.14-365-g451ceae1-1focal python3-cephfs=16.2.14-365-g451ceae1-1focal python3-rbd=16.2.14-365-g451ceae1-1focal libcephfs2=16.2.14-365-g451ceae1-1focal libcephfs-dev=16.2.14-365-g451ceae1-1focal librados2=16.2.14-365-g451ceae1-1focal librbd1=16.2.14-365-g451ceae1-1focal rbd-fuse=16.2.14-365-g451ceae1-1focal rbd-mirror=16.2.14-365-g451ceae1-1focal 2023-12-08T20:21:16.244 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-12-08T20:21:16.277 INFO:teuthology.orchestra.run.smithi158.stdout:Reading package lists... 2023-12-08T20:21:16.304 DEBUG:teuthology.orchestra.run.smithi158:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-365-g451ceae1-1focal cephadm=16.2.14-365-g451ceae1-1focal ceph-mds=16.2.14-365-g451ceae1-1focal ceph-mgr=16.2.14-365-g451ceae1-1focal ceph-common=16.2.14-365-g451ceae1-1focal ceph-fuse=16.2.14-365-g451ceae1-1focal ceph-test=16.2.14-365-g451ceae1-1focal radosgw=16.2.14-365-g451ceae1-1focal python3-rados=16.2.14-365-g451ceae1-1focal python3-rgw=16.2.14-365-g451ceae1-1focal python3-cephfs=16.2.14-365-g451ceae1-1focal python3-rbd=16.2.14-365-g451ceae1-1focal libcephfs2=16.2.14-365-g451ceae1-1focal libcephfs-dev=16.2.14-365-g451ceae1-1focal librados2=16.2.14-365-g451ceae1-1focal librbd1=16.2.14-365-g451ceae1-1focal rbd-fuse=16.2.14-365-g451ceae1-1focal rbd-mirror=16.2.14-365-g451ceae1-1focal 2023-12-08T20:21:16.375 INFO:teuthology.orchestra.run.smithi158.stdout:Reading package lists... 2023-12-08T20:21:16.426 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-12-08T20:21:16.427 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-12-08T20:21:16.579 INFO:teuthology.orchestra.run.smithi158.stdout:Building dependency tree... 2023-12-08T20:21:16.580 INFO:teuthology.orchestra.run.smithi158.stdout:Reading state information... 2023-12-08T20:21:16.597 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2023-12-08T20:21:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-08T20:21:16.598 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-08T20:21:16.598 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2023-12-08T20:21:16.598 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-08T20:21:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjaeger 2023-12-08T20:21:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2023-12-08T20:21:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2023-12-08T20:21:16.599 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli python-pastedeploy-tpl 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-12-08T20:21:16.600 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2023-12-08T20:21:16.601 INFO:teuthology.orchestra.run.smithi069.stdout: btrfs-tools 2023-12-08T20:21:16.705 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2023-12-08T20:21:16.706 INFO:teuthology.orchestra.run.smithi069.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-08T20:21:16.706 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-08T20:21:16.706 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2023-12-08T20:21:16.706 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2023-12-08T20:21:16.706 INFO:teuthology.orchestra.run.smithi069.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-12-08T20:21:16.707 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse rbd-mirror socat 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages will be upgraded: 2023-12-08T20:21:16.708 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 librbd1 2023-12-08T20:21:16.763 INFO:teuthology.orchestra.run.smithi158.stdout:The following packages were automatically installed and are no longer required: 2023-12-08T20:21:16.764 INFO:teuthology.orchestra.run.smithi158.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-08T20:21:16.765 INFO:teuthology.orchestra.run.smithi158.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-08T20:21:16.765 INFO:teuthology.orchestra.run.smithi158.stdout:The following additional packages will be installed: 2023-12-08T20:21:16.765 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-08T20:21:16.765 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjaeger 2023-12-08T20:21:16.765 INFO:teuthology.orchestra.run.smithi158.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2023-12-08T20:21:16.765 INFO:teuthology.orchestra.run.smithi158.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli python-pastedeploy-tpl 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-12-08T20:21:16.766 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout:Suggested packages: 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-12-08T20:21:16.767 INFO:teuthology.orchestra.run.smithi158.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-12-08T20:21:16.768 INFO:teuthology.orchestra.run.smithi158.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-08T20:21:16.768 INFO:teuthology.orchestra.run.smithi158.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-12-08T20:21:16.768 INFO:teuthology.orchestra.run.smithi158.stdout:Recommended packages: 2023-12-08T20:21:16.768 INFO:teuthology.orchestra.run.smithi158.stdout: btrfs-tools 2023-12-08T20:21:16.838 INFO:teuthology.orchestra.run.smithi069.stdout:2 upgraded, 99 newly installed, 0 to remove and 259 not upgraded. 2023-12-08T20:21:16.838 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 177 MB of archives. 2023-12-08T20:21:16.839 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 821 MB of additional disk space will be used. 2023-12-08T20:21:16.839 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.2.14-365-g451ceae1-1focal [3,960 B] 2023-12-08T20:21:16.877 INFO:teuthology.orchestra.run.smithi158.stdout:The following NEW packages will be installed: 2023-12-08T20:21:16.877 INFO:teuthology.orchestra.run.smithi158.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-08T20:21:16.877 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-08T20:21:16.877 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2023-12-08T20:21:16.877 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2023-12-08T20:21:16.877 INFO:teuthology.orchestra.run.smithi158.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2023-12-08T20:21:16.878 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli 2023-12-08T20:21:16.878 INFO:teuthology.orchestra.run.smithi158.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-12-08T20:21:16.878 INFO:teuthology.orchestra.run.smithi158.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse rbd-mirror socat 2023-12-08T20:21:16.879 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet 2023-12-08T20:21:16.880 INFO:teuthology.orchestra.run.smithi158.stdout:The following packages will be upgraded: 2023-12-08T20:21:16.880 INFO:teuthology.orchestra.run.smithi158.stdout: librados2 librbd1 2023-12-08T20:21:16.882 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.2.14-365-g451ceae1-1focal [3,386 kB] 2023-12-08T20:21:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-12-08T20:21:17.022 INFO:teuthology.orchestra.run.smithi158.stdout:2 upgraded, 99 newly installed, 0 to remove and 259 not upgraded. 2023-12-08T20:21:17.023 INFO:teuthology.orchestra.run.smithi158.stdout:Need to get 177 MB of archives. 2023-12-08T20:21:17.023 INFO:teuthology.orchestra.run.smithi158.stdout:After this operation, 821 MB of additional disk space will be used. 2023-12-08T20:21:17.023 INFO:teuthology.orchestra.run.smithi158.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.2.14-365-g451ceae1-1focal [3,960 B] 2023-12-08T20:21:17.067 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.2.14-365-g451ceae1-1focal [3,388 kB] 2023-12-08T20:21:17.071 INFO:teuthology.orchestra.run.smithi158.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-12-08T20:21:17.079 INFO:teuthology.orchestra.run.smithi158.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.2.14-365-g451ceae1-1focal [3,386 kB] 2023-12-08T20:21:17.186 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.2.14-365-g451ceae1-1focal [729 kB] 2023-12-08T20:21:17.210 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.2.14-365-g451ceae1-1focal [414 kB] 2023-12-08T20:21:17.225 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.2.14-365-g451ceae1-1focal [22.2 kB] 2023-12-08T20:21:17.227 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.2.14-365-g451ceae1-1focal [212 kB] 2023-12-08T20:21:17.233 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.2.14-365-g451ceae1-1focal [38.4 kB] 2023-12-08T20:21:17.235 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.2.14-365-g451ceae1-1focal [386 kB] 2023-12-08T20:21:17.248 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.2.14-365-g451ceae1-1focal [3,706 kB] 2023-12-08T20:21:17.305 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-12-08T20:21:17.313 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-12-08T20:21:17.356 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-12-08T20:21:17.379 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.2.14-365-g451ceae1-1focal [116 kB] 2023-12-08T20:21:17.385 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.2.14-365-g451ceae1-1focal [448 kB] 2023-12-08T20:21:17.386 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-12-08T20:21:17.400 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.2.14-365-g451ceae1-1focal [22.7 MB] 2023-12-08T20:21:17.467 INFO:teuthology.orchestra.run.smithi158.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-12-08T20:21:17.474 INFO:teuthology.orchestra.run.smithi158.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-12-08T20:21:17.496 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2023-12-08T20:21:17.519 INFO:teuthology.orchestra.run.smithi158.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-12-08T20:21:17.520 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-12-08T20:21:17.527 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2023-12-08T20:21:17.532 INFO:teuthology.orchestra.run.smithi158.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-12-08T20:21:17.558 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-12-08T20:21:17.562 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-12-08T20:21:17.565 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-12-08T20:21:17.659 INFO:teuthology.orchestra.run.smithi158.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2023-12-08T20:21:17.681 INFO:teuthology.orchestra.run.smithi158.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-12-08T20:21:17.688 INFO:teuthology.orchestra.run.smithi158.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2023-12-08T20:21:17.718 INFO:teuthology.orchestra.run.smithi158.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-12-08T20:21:17.723 INFO:teuthology.orchestra.run.smithi158.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-12-08T20:21:17.726 INFO:teuthology.orchestra.run.smithi158.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-12-08T20:21:17.744 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-12-08T20:21:17.905 INFO:teuthology.orchestra.run.smithi158.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-12-08T20:21:17.922 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-12-08T20:21:17.952 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-12-08T20:21:17.982 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-12-08T20:21:18.079 INFO:teuthology.orchestra.run.smithi158.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-12-08T20:21:18.099 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-08T20:21:18.109 INFO:teuthology.orchestra.run.smithi158.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-12-08T20:21:18.109 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-12-08T20:21:18.114 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-12-08T20:21:18.160 INFO:teuthology.orchestra.run.smithi158.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-12-08T20:21:18.182 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-12-08T20:21:18.199 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-12-08T20:21:18.233 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-12-08T20:21:18.259 INFO:teuthology.orchestra.run.smithi158.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-08T20:21:18.269 INFO:teuthology.orchestra.run.smithi158.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-12-08T20:21:18.274 INFO:teuthology.orchestra.run.smithi158.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-12-08T20:21:18.309 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-12-08T20:21:18.310 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.2.14-365-g451ceae1-1focal [5,917 kB] 2023-12-08T20:21:18.343 INFO:teuthology.orchestra.run.smithi158.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-12-08T20:21:18.359 INFO:teuthology.orchestra.run.smithi158.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-12-08T20:21:18.394 INFO:teuthology.orchestra.run.smithi158.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-12-08T20:21:18.423 INFO:teuthology.orchestra.run.smithi158.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-12-08T20:21:18.425 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.2.14-365-g451ceae1-1focal [204 kB] 2023-12-08T20:21:18.428 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 16.2.14-365-g451ceae1-1focal [126 kB] 2023-12-08T20:21:18.433 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.2.14-365-g451ceae1-1focal [1,551 kB] 2023-12-08T20:21:18.474 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-12-08T20:21:18.492 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.2.14-365-g451ceae1-1focal [6,997 kB] 2023-12-08T20:21:18.538 INFO:teuthology.orchestra.run.smithi158.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.2.14-365-g451ceae1-1focal [3,388 kB] 2023-12-08T20:21:18.588 INFO:teuthology.orchestra.run.smithi158.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-12-08T20:21:18.746 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.2.14-365-g451ceae1-1focal [26.2 MB] 2023-12-08T20:21:19.011 INFO:teuthology.orchestra.run.smithi069.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-12-08T20:21:19.012 INFO:teuthology.orchestra.run.smithi069.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-12-08T20:21:19.015 INFO:teuthology.orchestra.run.smithi069.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-12-08T20:21:19.018 INFO:teuthology.orchestra.run.smithi069.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-12-08T20:21:19.030 INFO:teuthology.orchestra.run.smithi069.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-12-08T20:21:19.053 INFO:teuthology.orchestra.run.smithi069.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-12-08T20:21:19.066 INFO:teuthology.orchestra.run.smithi069.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-08T20:21:19.067 INFO:teuthology.orchestra.run.smithi069.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-12-08T20:21:19.103 INFO:teuthology.orchestra.run.smithi069.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-12-08T20:21:19.114 INFO:teuthology.orchestra.run.smithi069.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-12-08T20:21:19.121 INFO:teuthology.orchestra.run.smithi158.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-12-08T20:21:19.122 INFO:teuthology.orchestra.run.smithi158.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-12-08T20:21:19.125 INFO:teuthology.orchestra.run.smithi158.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-12-08T20:21:19.129 INFO:teuthology.orchestra.run.smithi158.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-12-08T20:21:19.140 INFO:teuthology.orchestra.run.smithi158.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-12-08T20:21:19.163 INFO:teuthology.orchestra.run.smithi158.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-12-08T20:21:19.175 INFO:teuthology.orchestra.run.smithi158.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-08T20:21:19.176 INFO:teuthology.orchestra.run.smithi158.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-12-08T20:21:19.212 INFO:teuthology.orchestra.run.smithi158.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-12-08T20:21:19.223 INFO:teuthology.orchestra.run.smithi158.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-12-08T20:21:19.290 INFO:teuthology.orchestra.run.smithi069.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-12-08T20:21:19.375 INFO:teuthology.orchestra.run.smithi069.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-12-08T20:21:19.402 INFO:teuthology.orchestra.run.smithi158.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-12-08T20:21:19.458 INFO:teuthology.orchestra.run.smithi069.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-12-08T20:21:19.482 INFO:teuthology.orchestra.run.smithi158.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-12-08T20:21:19.566 INFO:teuthology.orchestra.run.smithi158.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-12-08T20:21:19.782 INFO:teuthology.orchestra.run.smithi158.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.2.14-365-g451ceae1-1focal [729 kB] 2023-12-08T20:21:19.862 INFO:teuthology.orchestra.run.smithi069.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.2.14-365-g451ceae1-1focal [4,048 B] 2023-12-08T20:21:19.863 INFO:teuthology.orchestra.run.smithi069.stdout:Get:57 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.2.14-365-g451ceae1-1focal [835 kB] 2023-12-08T20:21:19.876 INFO:teuthology.orchestra.run.smithi069.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.2.14-365-g451ceae1-1focal [2,364 kB] 2023-12-08T20:21:19.880 INFO:teuthology.orchestra.run.smithi158.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.2.14-365-g451ceae1-1focal [414 kB] 2023-12-08T20:21:19.894 INFO:teuthology.orchestra.run.smithi069.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.2.14-365-g451ceae1-1focal [77.4 kB] 2023-12-08T20:21:19.894 INFO:teuthology.orchestra.run.smithi069.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.2.14-365-g451ceae1-1focal [100 kB] 2023-12-08T20:21:19.895 INFO:teuthology.orchestra.run.smithi069.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.2.14-365-g451ceae1-1focal [2,824 kB] 2023-12-08T20:21:19.955 INFO:teuthology.orchestra.run.smithi069.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.2.14-365-g451ceae1-1focal [7,884 kB] 2023-12-08T20:21:20.024 INFO:teuthology.orchestra.run.smithi158.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.2.14-365-g451ceae1-1focal [22.2 kB] 2023-12-08T20:21:20.025 INFO:teuthology.orchestra.run.smithi158.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.2.14-365-g451ceae1-1focal [212 kB] 2023-12-08T20:21:20.087 INFO:teuthology.orchestra.run.smithi158.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.2.14-365-g451ceae1-1focal [38.4 kB] 2023-12-08T20:21:20.088 INFO:teuthology.orchestra.run.smithi158.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.2.14-365-g451ceae1-1focal [386 kB] 2023-12-08T20:21:20.201 INFO:teuthology.orchestra.run.smithi069.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-12-08T20:21:20.214 INFO:teuthology.orchestra.run.smithi158.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.2.14-365-g451ceae1-1focal [3,706 kB] 2023-12-08T20:21:20.240 INFO:teuthology.orchestra.run.smithi069.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.2.14-365-g451ceae1-1focal [4,164 B] 2023-12-08T20:21:20.240 INFO:teuthology.orchestra.run.smithi069.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.2.14-365-g451ceae1-1focal [49.4 MB] 2023-12-08T20:21:20.278 INFO:teuthology.orchestra.run.smithi069.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-12-08T20:21:20.280 INFO:teuthology.orchestra.run.smithi069.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-12-08T20:21:20.309 INFO:teuthology.orchestra.run.smithi158.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-12-08T20:21:20.328 INFO:teuthology.orchestra.run.smithi069.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-12-08T20:21:20.329 INFO:teuthology.orchestra.run.smithi069.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-12-08T20:21:20.330 INFO:teuthology.orchestra.run.smithi069.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-12-08T20:21:20.331 INFO:teuthology.orchestra.run.smithi069.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-12-08T20:21:20.331 INFO:teuthology.orchestra.run.smithi069.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-12-08T20:21:20.382 INFO:teuthology.orchestra.run.smithi158.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-12-08T20:21:20.385 INFO:teuthology.orchestra.run.smithi158.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-12-08T20:21:20.432 INFO:teuthology.orchestra.run.smithi158.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-12-08T20:21:20.432 INFO:teuthology.orchestra.run.smithi158.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-12-08T20:21:20.434 INFO:teuthology.orchestra.run.smithi158.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-12-08T20:21:20.434 INFO:teuthology.orchestra.run.smithi158.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-12-08T20:21:20.435 INFO:teuthology.orchestra.run.smithi158.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-12-08T20:21:20.495 INFO:teuthology.orchestra.run.smithi069.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-12-08T20:21:20.600 INFO:teuthology.orchestra.run.smithi158.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-12-08T20:21:20.661 INFO:teuthology.orchestra.run.smithi069.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-12-08T20:21:20.683 INFO:teuthology.orchestra.run.smithi158.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-12-08T20:21:20.847 INFO:teuthology.orchestra.run.smithi069.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-12-08T20:21:20.862 INFO:teuthology.orchestra.run.smithi069.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-12-08T20:21:20.863 INFO:teuthology.orchestra.run.smithi158.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-12-08T20:21:20.879 INFO:teuthology.orchestra.run.smithi158.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-12-08T20:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-12-08T20:21:20.911 INFO:teuthology.orchestra.run.smithi158.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-12-08T20:21:21.026 INFO:teuthology.orchestra.run.smithi069.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-12-08T20:21:21.041 INFO:teuthology.orchestra.run.smithi158.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-12-08T20:21:21.062 INFO:teuthology.orchestra.run.smithi069.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-12-08T20:21:21.077 INFO:teuthology.orchestra.run.smithi158.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-12-08T20:21:21.085 INFO:teuthology.orchestra.run.smithi069.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-12-08T20:21:21.094 INFO:teuthology.orchestra.run.smithi069.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-12-08T20:21:21.100 INFO:teuthology.orchestra.run.smithi158.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-12-08T20:21:21.109 INFO:teuthology.orchestra.run.smithi158.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-12-08T20:21:21.137 INFO:teuthology.orchestra.run.smithi069.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-12-08T20:21:21.152 INFO:teuthology.orchestra.run.smithi158.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-12-08T20:21:21.163 INFO:teuthology.orchestra.run.smithi069.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-12-08T20:21:21.178 INFO:teuthology.orchestra.run.smithi158.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-12-08T20:21:21.350 INFO:teuthology.orchestra.run.smithi069.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-12-08T20:21:21.364 INFO:teuthology.orchestra.run.smithi158.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-12-08T20:21:21.433 INFO:teuthology.orchestra.run.smithi069.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-12-08T20:21:21.448 INFO:teuthology.orchestra.run.smithi158.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-12-08T20:21:21.541 INFO:teuthology.orchestra.run.smithi069.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-12-08T20:21:21.549 INFO:teuthology.orchestra.run.smithi158.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-12-08T20:21:21.563 INFO:teuthology.orchestra.run.smithi069.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-12-08T20:21:21.572 INFO:teuthology.orchestra.run.smithi158.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-12-08T20:21:21.603 INFO:teuthology.orchestra.run.smithi158.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.2.14-365-g451ceae1-1focal [116 kB] 2023-12-08T20:21:21.670 INFO:teuthology.orchestra.run.smithi158.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.2.14-365-g451ceae1-1focal [448 kB] 2023-12-08T20:21:21.682 INFO:teuthology.orchestra.run.smithi069.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-12-08T20:21:21.686 INFO:teuthology.orchestra.run.smithi158.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-12-08T20:21:21.871 INFO:teuthology.orchestra.run.smithi158.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.2.14-365-g451ceae1-1focal [22.7 MB] 2023-12-08T20:21:21.966 INFO:teuthology.orchestra.run.smithi069.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-12-08T20:21:21.967 INFO:teuthology.orchestra.run.smithi069.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-08T20:21:21.971 INFO:teuthology.orchestra.run.smithi158.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-12-08T20:21:21.972 INFO:teuthology.orchestra.run.smithi069.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-12-08T20:21:21.973 INFO:teuthology.orchestra.run.smithi069.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-12-08T20:21:21.973 INFO:teuthology.orchestra.run.smithi158.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-08T20:21:21.977 INFO:teuthology.orchestra.run.smithi158.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-12-08T20:21:21.977 INFO:teuthology.orchestra.run.smithi158.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-12-08T20:21:21.982 INFO:teuthology.orchestra.run.smithi069.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-12-08T20:21:21.987 INFO:teuthology.orchestra.run.smithi069.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-12-08T20:21:21.988 INFO:teuthology.orchestra.run.smithi158.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-12-08T20:21:21.993 INFO:teuthology.orchestra.run.smithi158.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-12-08T20:21:22.152 INFO:teuthology.orchestra.run.smithi069.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-12-08T20:21:22.157 INFO:teuthology.orchestra.run.smithi158.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-12-08T20:21:22.230 INFO:teuthology.orchestra.run.smithi069.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.2.14-365-g451ceae1-1focal [21.5 kB] 2023-12-08T20:21:22.231 INFO:teuthology.orchestra.run.smithi069.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.2.14-365-g451ceae1-1focal [11.3 MB] 2023-12-08T20:21:22.320 INFO:teuthology.orchestra.run.smithi069.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-12-08T20:21:22.325 INFO:teuthology.orchestra.run.smithi158.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-12-08T20:21:22.558 INFO:teuthology.orchestra.run.smithi069.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.2.14-365-g451ceae1-1focal [78.6 kB] 2023-12-08T20:21:22.559 INFO:teuthology.orchestra.run.smithi069.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 rbd-mirror amd64 16.2.14-365-g451ceae1-1focal [3,468 kB] 2023-12-08T20:21:22.571 INFO:teuthology.orchestra.run.smithi069.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-12-08T20:21:22.577 INFO:teuthology.orchestra.run.smithi158.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-12-08T20:21:23.261 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 177 MB in 6s (29.9 MB/s) 2023-12-08T20:21:23.354 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjaeger. 2023-12-08T20:21:23.411 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141731 files and directories currently installed.) 2023-12-08T20:21:23.415 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../000-libjaeger_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:23.460 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjaeger (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:23.763 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-12-08T20:21:23.781 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-12-08T20:21:23.846 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-08T20:21:24.124 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../002-librbd1_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:24.284 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librbd1 (16.2.14-365-g451ceae1-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-08T20:21:24.637 INFO:teuthology.orchestra.run.smithi158.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.2.14-365-g451ceae1-1focal [5,917 kB] 2023-12-08T20:21:24.995 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../003-librados2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:25.077 INFO:teuthology.orchestra.run.smithi158.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.2.14-365-g451ceae1-1focal [204 kB] 2023-12-08T20:21:25.095 INFO:teuthology.orchestra.run.smithi158.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 16.2.14-365-g451ceae1-1focal [126 kB] 2023-12-08T20:21:25.112 INFO:teuthology.orchestra.run.smithi158.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.2.14-365-g451ceae1-1focal [1,551 kB] 2023-12-08T20:21:25.121 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librados2 (16.2.14-365-g451ceae1-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-08T20:21:25.221 INFO:teuthology.orchestra.run.smithi158.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.2.14-365-g451ceae1-1focal [6,997 kB] 2023-12-08T20:21:25.738 INFO:teuthology.orchestra.run.smithi158.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.2.14-365-g451ceae1-1focal [26.2 MB] 2023-12-08T20:21:25.742 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs2. 2023-12-08T20:21:25.758 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../004-libcephfs2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:25.800 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:26.153 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rados. 2023-12-08T20:21:26.169 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../005-python3-rados_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:26.211 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rados (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:26.513 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-08T20:21:26.530 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:26.563 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-argparse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:26.900 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cephfs. 2023-12-08T20:21:26.919 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../007-python3-cephfs_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:26.957 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cephfs (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:27.320 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-08T20:21:27.337 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../008-python3-ceph-common_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:27.378 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:27.538 INFO:teuthology.orchestra.run.smithi158.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.2.14-365-g451ceae1-1focal [4,048 B] 2023-12-08T20:21:27.538 INFO:teuthology.orchestra.run.smithi158.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.2.14-365-g451ceae1-1focal [835 kB] 2023-12-08T20:21:27.560 INFO:teuthology.orchestra.run.smithi158.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.2.14-365-g451ceae1-1focal [2,364 kB] 2023-12-08T20:21:27.688 INFO:teuthology.orchestra.run.smithi158.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.2.14-365-g451ceae1-1focal [77.4 kB] 2023-12-08T20:21:27.704 INFO:teuthology.orchestra.run.smithi158.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.2.14-365-g451ceae1-1focal [100 kB] 2023-12-08T20:21:27.705 INFO:teuthology.orchestra.run.smithi158.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.2.14-365-g451ceae1-1focal [2,824 kB] 2023-12-08T20:21:27.706 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-prettytable. 2023-12-08T20:21:27.722 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2023-12-08T20:21:27.764 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-12-08T20:21:27.871 INFO:teuthology.orchestra.run.smithi158.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.2.14-365-g451ceae1-1focal [7,884 kB] 2023-12-08T20:21:28.017 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rbd. 2023-12-08T20:21:28.033 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../010-python3-rbd_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:28.100 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rbd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:28.309 INFO:teuthology.orchestra.run.smithi158.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.2.14-365-g451ceae1-1focal [4,164 B] 2023-12-08T20:21:28.309 INFO:teuthology.orchestra.run.smithi158.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.2.14-365-g451ceae1-1focal [49.4 MB] 2023-12-08T20:21:28.470 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-12-08T20:21:28.486 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-12-08T20:21:28.553 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-08T20:21:28.889 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-12-08T20:21:28.908 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2023-12-08T20:21:28.947 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-12-08T20:21:29.317 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-08T20:21:29.336 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-12-08T20:21:29.375 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-08T20:21:29.736 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librgw2. 2023-12-08T20:21:29.755 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../014-librgw2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:29.794 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librgw2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:30.372 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rgw. 2023-12-08T20:21:30.389 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../015-python3-rgw_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:30.421 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:30.766 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libleveldb1d:amd64. 2023-12-08T20:21:30.782 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2023-12-08T20:21:30.824 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-12-08T20:21:31.000 INFO:teuthology.orchestra.run.smithi158.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.2.14-365-g451ceae1-1focal [21.5 kB] 2023-12-08T20:21:31.000 INFO:teuthology.orchestra.run.smithi158.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.2.14-365-g451ceae1-1focal [11.3 MB] 2023-12-08T20:21:31.211 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liboath0. 2023-12-08T20:21:31.231 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2023-12-08T20:21:31.269 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-12-08T20:21:31.481 INFO:teuthology.orchestra.run.smithi158.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.2.14-365-g451ceae1-1focal [78.6 kB] 2023-12-08T20:21:31.483 INFO:teuthology.orchestra.run.smithi158.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 rbd-mirror amd64 16.2.14-365-g451ceae1-1focal [3,468 kB] 2023-12-08T20:21:31.588 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libradosstriper1. 2023-12-08T20:21:31.604 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../018-libradosstriper1_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:31.638 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libradosstriper1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:32.108 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-common. 2023-12-08T20:21:32.125 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../019-ceph-common_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:32.166 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:32.228 INFO:teuthology.orchestra.run.smithi158.stdout:Fetched 177 MB in 15s (12.0 MB/s) 2023-12-08T20:21:32.322 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libjaeger. 2023-12-08T20:21:32.378 INFO:teuthology.orchestra.run.smithi158.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141731 files and directories currently installed.) 2023-12-08T20:21:32.382 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../000-libjaeger_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:32.420 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libjaeger (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:32.724 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-12-08T20:21:32.741 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-12-08T20:21:32.824 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-08T20:21:33.168 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../002-librbd1_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:33.294 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking librbd1 (16.2.14-365-g451ceae1-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-08T20:21:34.240 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../003-librados2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:34.349 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking librados2 (16.2.14-365-g451ceae1-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-08T20:21:34.405 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-base. 2023-12-08T20:21:34.421 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../020-ceph-base_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:34.490 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-base (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:35.027 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libcephfs2. 2023-12-08T20:21:35.048 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../004-libcephfs2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:35.086 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libcephfs2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:35.422 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-rados. 2023-12-08T20:21:35.440 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../005-python3-rados_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:35.480 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-rados (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:35.509 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cryptography. 2023-12-08T20:21:35.526 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2023-12-08T20:21:35.567 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-08T20:21:35.800 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-08T20:21:35.819 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:35.858 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-ceph-argparse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:35.971 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-openssl. 2023-12-08T20:21:35.991 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2023-12-08T20:21:36.030 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-12-08T20:21:36.186 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-cephfs. 2023-12-08T20:21:36.204 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../007-python3-cephfs_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:36.245 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-cephfs (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:36.273 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-08T20:21:36.292 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:36.323 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-modules-core (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:36.581 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-08T20:21:36.597 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../008-python3-ceph-common_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:36.639 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:36.728 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-08T20:21:36.745 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-12-08T20:21:36.786 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-08T20:21:36.950 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-prettytable. 2023-12-08T20:21:36.968 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2023-12-08T20:21:37.008 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-12-08T20:21:37.088 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-08T20:21:37.105 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-12-08T20:21:37.147 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-12-08T20:21:37.261 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-rbd. 2023-12-08T20:21:37.277 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../010-python3-rbd_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:37.310 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-rbd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:37.517 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jwt. 2023-12-08T20:21:37.534 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-12-08T20:21:37.567 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-08T20:21:37.639 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-12-08T20:21:37.658 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-12-08T20:21:37.697 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-08T20:21:37.828 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-logutils. 2023-12-08T20:21:37.848 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-12-08T20:21:37.878 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-12-08T20:21:38.050 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-12-08T20:21:38.065 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2023-12-08T20:21:38.108 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-12-08T20:21:38.180 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-08T20:21:38.200 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-12-08T20:21:38.239 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-12-08T20:21:38.410 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-08T20:21:38.428 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-12-08T20:21:38.460 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-08T20:21:38.566 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-mako. 2023-12-08T20:21:38.582 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-12-08T20:21:38.633 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-08T20:21:38.789 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package librgw2. 2023-12-08T20:21:38.807 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../014-librgw2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:38.847 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking librgw2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:38.978 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-08T20:21:38.994 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-08T20:21:39.036 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-08T20:21:39.305 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-08T20:21:39.324 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-12-08T20:21:39.355 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-12-08T20:21:39.483 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-rgw. 2023-12-08T20:21:39.500 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../015-python3-rgw_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:39.541 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-rgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:39.633 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webob. 2023-12-08T20:21:39.649 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-12-08T20:21:39.691 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-12-08T20:21:39.861 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libleveldb1d:amd64. 2023-12-08T20:21:39.877 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2023-12-08T20:21:39.919 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-12-08T20:21:40.027 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-soupsieve. 2023-12-08T20:21:40.044 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-12-08T20:21:40.085 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-08T20:21:40.239 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package liboath0. 2023-12-08T20:21:40.258 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2023-12-08T20:21:40.297 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-12-08T20:21:40.388 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-bs4. 2023-12-08T20:21:40.408 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-12-08T20:21:40.446 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-12-08T20:21:40.641 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libradosstriper1. 2023-12-08T20:21:40.659 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../018-libradosstriper1_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:40.699 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libradosstriper1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:40.783 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-waitress. 2023-12-08T20:21:40.799 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-12-08T20:21:40.861 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-08T20:21:41.069 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-common. 2023-12-08T20:21:41.087 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../019-ceph-common_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:41.127 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:41.211 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-tempita. 2023-12-08T20:21:41.229 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-12-08T20:21:41.269 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-12-08T20:21:41.555 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-paste. 2023-12-08T20:21:41.571 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-12-08T20:21:41.613 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-12-08T20:21:41.958 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-08T20:21:41.976 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-12-08T20:21:42.016 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-12-08T20:21:42.327 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-08T20:21:42.343 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-12-08T20:21:42.377 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-12-08T20:21:42.671 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webtest. 2023-12-08T20:21:42.687 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-12-08T20:21:42.763 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-12-08T20:21:43.074 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pecan. 2023-12-08T20:21:43.090 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-12-08T20:21:43.132 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-12-08T20:21:43.358 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-base. 2023-12-08T20:21:43.374 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../020-ceph-base_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:43.427 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-base (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:43.512 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-08T20:21:43.529 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-12-08T20:21:43.570 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-08T20:21:43.959 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-08T20:21:43.976 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../043-libsqlite3-mod-ceph_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:44.015 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libsqlite3-mod-ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:44.337 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-cryptography. 2023-12-08T20:21:44.355 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2023-12-08T20:21:44.387 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-08T20:21:44.393 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr. 2023-12-08T20:21:44.410 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../044-ceph-mgr_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:44.451 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:44.724 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-openssl. 2023-12-08T20:21:44.740 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2023-12-08T20:21:44.774 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-12-08T20:21:44.971 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mon. 2023-12-08T20:21:44.987 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../045-ceph-mon_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:45.029 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mon (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:45.043 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-08T20:21:45.060 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:45.101 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mgr-modules-core (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:45.473 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-08T20:21:45.489 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-12-08T20:21:45.531 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-08T20:21:45.834 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-08T20:21:45.850 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-12-08T20:21:45.892 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-12-08T20:21:46.133 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-osd. 2023-12-08T20:21:46.152 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../046-ceph-osd_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:46.191 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-osd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:46.279 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-jwt. 2023-12-08T20:21:46.296 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-12-08T20:21:46.337 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-08T20:21:46.649 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-logutils. 2023-12-08T20:21:46.665 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-12-08T20:21:46.707 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-12-08T20:21:47.010 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-08T20:21:47.026 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-12-08T20:21:47.068 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-12-08T20:21:47.396 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-mako. 2023-12-08T20:21:47.412 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-12-08T20:21:47.454 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-08T20:21:47.774 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-08T20:21:47.790 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-08T20:21:47.832 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-08T20:21:48.135 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-08T20:21:48.152 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-12-08T20:21:48.193 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-12-08T20:21:48.529 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-webob. 2023-12-08T20:21:48.547 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-12-08T20:21:48.587 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-12-08T20:21:48.621 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph. 2023-12-08T20:21:48.641 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../047-ceph_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:48.704 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:48.890 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-soupsieve. 2023-12-08T20:21:48.910 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-12-08T20:21:48.948 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-08T20:21:49.023 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-fuse. 2023-12-08T20:21:49.044 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../048-ceph-fuse_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:49.235 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-bs4. 2023-12-08T20:21:49.254 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-12-08T20:21:49.340 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:49.543 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-12-08T20:21:50.363 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-waitress. 2023-12-08T20:21:50.380 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-12-08T20:21:50.424 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-08T20:21:50.485 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mds. 2023-12-08T20:21:50.504 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../049-ceph-mds_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:50.543 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mds (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:50.774 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-tempita. 2023-12-08T20:21:50.792 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-12-08T20:21:50.832 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-12-08T20:21:51.071 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package cephadm. 2023-12-08T20:21:51.090 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../050-cephadm_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:51.129 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:51.135 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-paste. 2023-12-08T20:21:51.154 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-12-08T20:21:51.193 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-12-08T20:21:51.432 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jinja2. 2023-12-08T20:21:51.450 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2023-12-08T20:21:51.490 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-12-08T20:21:51.505 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-08T20:21:51.522 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-12-08T20:21:51.571 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-12-08T20:21:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-08T20:21:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:51.899 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-08T20:21:51.916 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-12-08T20:21:51.957 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-12-08T20:21:52.196 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-08T20:21:52.216 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2023-12-08T20:21:52.254 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-08T20:21:52.269 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-webtest. 2023-12-08T20:21:52.285 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-12-08T20:21:52.327 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-12-08T20:21:52.506 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjs-underscore. 2023-12-08T20:21:52.523 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-12-08T20:21:52.565 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-08T20:21:52.638 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pecan. 2023-12-08T20:21:52.654 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-12-08T20:21:52.696 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-12-08T20:21:52.859 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-12-08T20:21:52.875 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-12-08T20:21:52.917 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-08T20:21:53.051 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-08T20:21:53.070 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-12-08T20:21:53.109 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-08T20:21:53.245 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-routes. 2023-12-08T20:21:53.262 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2023-12-08T20:21:53.311 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-12-08T20:21:53.455 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-08T20:21:53.474 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../043-libsqlite3-mod-ceph_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:53.512 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libsqlite3-mod-ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:53.647 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-08T20:21:53.665 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:53.705 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-dashboard (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:53.824 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mgr. 2023-12-08T20:21:53.840 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../044-ceph-mgr_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:53.874 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mgr (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:54.385 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mon. 2023-12-08T20:21:54.401 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../045-ceph-mon_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:54.443 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mon (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:54.666 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-decorator. 2023-12-08T20:21:54.685 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../058-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-12-08T20:21:54.744 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-12-08T20:21:55.051 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-12-08T20:21:55.067 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../059-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-12-08T20:21:55.109 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-08T20:21:55.420 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-scipy. 2023-12-08T20:21:55.438 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../060-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-12-08T20:21:55.470 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-12-08T20:21:55.580 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-osd. 2023-12-08T20:21:55.597 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../046-ceph-osd_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:55.638 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-osd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:56.796 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-12-08T20:21:56.814 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-12-08T20:21:56.854 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-08T20:21:57.407 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-joblib. 2023-12-08T20:21:57.425 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../062-python3-joblib_0.14.0-3_all.deb ... 2023-12-08T20:21:57.457 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-12-08T20:21:57.852 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn. 2023-12-08T20:21:57.869 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../063-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-12-08T20:21:57.910 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-08T20:21:58.310 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph. 2023-12-08T20:21:58.326 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../047-ceph_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:58.368 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:58.618 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-08T20:21:58.635 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:21:58.676 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:58.713 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-fuse. 2023-12-08T20:21:58.729 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../048-ceph-fuse_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:58.770 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:59.182 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mds. 2023-12-08T20:21:59.200 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../049-ceph-mds_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:59.241 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mds (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:59.777 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package cephadm. 2023-12-08T20:21:59.795 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../050-cephadm_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:21:59.796 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cachetools. 2023-12-08T20:21:59.814 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../065-python3-cachetools_4.0.0-1_all.deb ... 2023-12-08T20:21:59.835 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:21:59.854 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-12-08T20:22:00.105 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-jinja2. 2023-12-08T20:22:00.121 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2023-12-08T20:22:00.159 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-12-08T20:22:00.182 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-08T20:22:00.199 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../066-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-12-08T20:22:00.240 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-12-08T20:22:00.482 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-08T20:22:00.496 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:22:00.537 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mgr-cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:00.585 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-08T20:22:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../067-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-12-08T20:22:00.643 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-08T20:22:00.899 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-08T20:22:00.918 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2023-12-08T20:22:00.951 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rsa. 2023-12-08T20:22:00.957 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-08T20:22:00.969 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../068-python3-rsa_4.0-3_all.deb ... 2023-12-08T20:22:01.035 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rsa (4.0-3) ... 2023-12-08T20:22:01.210 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libjs-underscore. 2023-12-08T20:22:01.228 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-12-08T20:22:01.268 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-08T20:22:01.355 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-google-auth. 2023-12-08T20:22:01.371 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../069-python3-google-auth_1.5.1-2_all.deb ... 2023-12-08T20:22:01.414 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-12-08T20:22:01.529 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-12-08T20:22:01.545 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-12-08T20:22:01.587 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-08T20:22:01.766 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-blinker. 2023-12-08T20:22:01.783 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../070-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-12-08T20:22:01.824 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-08T20:22:01.939 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-routes. 2023-12-08T20:22:01.956 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2023-12-08T20:22:01.998 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-12-08T20:22:02.152 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-oauthlib. 2023-12-08T20:22:02.168 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../071-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-12-08T20:22:02.210 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-08T20:22:02.342 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-08T20:22:02.360 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:22:02.409 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mgr-dashboard (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:02.580 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-08T20:22:02.598 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../072-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-12-08T20:22:02.638 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-08T20:22:02.966 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-websocket. 2023-12-08T20:22:02.986 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../073-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-12-08T20:22:03.024 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-12-08T20:22:03.344 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-08T20:22:03.362 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../074-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-12-08T20:22:03.384 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-decorator. 2023-12-08T20:22:03.401 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../058-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-12-08T20:22:03.427 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-12-08T20:22:03.443 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-12-08T20:22:03.746 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-12-08T20:22:03.762 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../059-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-12-08T20:22:03.804 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-08T20:22:03.974 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-08T20:22:03.994 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:22:04.034 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-k8sevents (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:04.148 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-scipy. 2023-12-08T20:22:04.167 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../060-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-12-08T20:22:04.198 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-12-08T20:22:04.389 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-08T20:22:04.409 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../076-libonig5_6.9.4-1_amd64.deb ... 2023-12-08T20:22:04.447 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-12-08T20:22:04.808 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-08T20:22:04.826 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../077-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-08T20:22:04.891 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:05.168 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package jq. 2023-12-08T20:22:05.184 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../078-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-08T20:22:05.226 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:05.504 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package socat. 2023-12-08T20:22:05.520 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../079-socat_1.7.3.3-2_amd64.deb ... 2023-12-08T20:22:05.562 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking socat (1.7.3.3-2) ... 2023-12-08T20:22:05.882 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package xmlstarlet. 2023-12-08T20:22:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../080-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-12-08T20:22:05.933 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-12-08T20:22:05.940 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-12-08T20:22:05.950 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-12-08T20:22:05.991 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-08T20:22:06.372 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-test. 2023-12-08T20:22:06.390 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../081-ceph-test_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:06.422 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-test (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:06.545 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-joblib. 2023-12-08T20:22:06.562 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../062-python3-joblib_0.14.0-3_all.deb ... 2023-12-08T20:22:06.603 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-12-08T20:22:06.957 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-sklearn. 2023-12-08T20:22:06.975 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../063-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-12-08T20:22:07.007 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-08T20:22:07.774 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-08T20:22:07.795 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:22:07.832 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:09.010 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-cachetools. 2023-12-08T20:22:09.028 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../065-python3-cachetools_4.0.0-1_all.deb ... 2023-12-08T20:22:09.068 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-12-08T20:22:09.346 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-08T20:22:09.366 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../066-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-12-08T20:22:09.404 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-12-08T20:22:09.733 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-08T20:22:09.751 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../067-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-12-08T20:22:09.816 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-08T20:22:10.111 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-rsa. 2023-12-08T20:22:10.131 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../068-python3-rsa_4.0-3_all.deb ... 2023-12-08T20:22:10.178 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-rsa (4.0-3) ... 2023-12-08T20:22:10.489 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-google-auth. 2023-12-08T20:22:10.505 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../069-python3-google-auth_1.5.1-2_all.deb ... 2023-12-08T20:22:10.547 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-12-08T20:22:10.801 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-08T20:22:10.818 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../082-libcephfs-dev_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:10.860 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs-dev (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:11.010 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-blinker. 2023-12-08T20:22:11.011 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../070-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-12-08T20:22:11.011 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-08T20:22:11.274 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-oauthlib. 2023-12-08T20:22:11.279 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package nvme-cli. 2023-12-08T20:22:11.290 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../071-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-12-08T20:22:11.295 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../083-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-12-08T20:22:11.328 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-08T20:22:11.363 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-12-08T20:22:11.688 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-08T20:22:11.690 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../072-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-12-08T20:22:11.731 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-08T20:22:11.760 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-atomicwrites. 2023-12-08T20:22:11.777 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../084-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-12-08T20:22:11.818 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-12-08T20:22:12.077 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-websocket. 2023-12-08T20:22:12.094 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../073-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-12-08T20:22:12.135 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-12-08T20:22:12.138 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-attr. 2023-12-08T20:22:12.157 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../085-python3-attr_19.3.0-2_all.deb ... 2023-12-08T20:22:12.197 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-12-08T20:22:12.463 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-08T20:22:12.480 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../074-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-12-08T20:22:12.522 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-12-08T20:22:12.590 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webencodings. 2023-12-08T20:22:12.607 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../086-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-12-08T20:22:12.648 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-08T20:22:12.959 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-html5lib. 2023-12-08T20:22:12.975 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../087-python3-html5lib_1.0.1-2_all.deb ... 2023-12-08T20:22:13.017 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-12-08T20:22:13.189 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-08T20:22:13.208 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-08T20:22:13.247 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-mgr-k8sevents (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:13.371 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-12-08T20:22:13.387 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../088-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-12-08T20:22:13.429 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-08T20:22:13.566 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-08T20:22:13.584 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../076-libonig5_6.9.4-1_amd64.deb ... 2023-12-08T20:22:13.624 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-12-08T20:22:13.890 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-packaging. 2023-12-08T20:22:13.907 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../089-python3-packaging_20.3-1_all.deb ... 2023-12-08T20:22:13.948 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-packaging (20.3-1) ... 2023-12-08T20:22:13.985 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-08T20:22:14.002 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../077-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-08T20:22:14.043 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:14.293 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastescript. 2023-12-08T20:22:14.310 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../090-python3-pastescript_2.0.2-4_all.deb ... 2023-12-08T20:22:14.321 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package jq. 2023-12-08T20:22:14.340 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../078-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-08T20:22:14.351 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-08T20:22:14.379 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:14.632 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package socat. 2023-12-08T20:22:14.649 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../079-socat_1.7.3.3-2_amd64.deb ... 2023-12-08T20:22:14.690 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking socat (1.7.3.3-2) ... 2023-12-08T20:22:14.714 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pluggy. 2023-12-08T20:22:14.731 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../091-python3-pluggy_0.13.0-2_all.deb ... 2023-12-08T20:22:14.773 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-12-08T20:22:14.985 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package xmlstarlet. 2023-12-08T20:22:15.001 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../080-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-12-08T20:22:15.043 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-12-08T20:22:15.100 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-psutil. 2023-12-08T20:22:15.120 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../092-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-12-08T20:22:15.159 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-12-08T20:22:15.493 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package ceph-test. 2023-12-08T20:22:15.510 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../081-ceph-test_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:15.528 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-py. 2023-12-08T20:22:15.549 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../093-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-12-08T20:22:15.550 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking ceph-test (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:15.586 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-12-08T20:22:15.914 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-08T20:22:15.933 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../094-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-12-08T20:22:15.989 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-08T20:22:16.300 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-08T20:22:16.317 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../095-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-12-08T20:22:16.358 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-08T20:22:16.686 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pytest. 2023-12-08T20:22:16.705 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../096-python3-pytest_4.6.9-1_all.deb ... 2023-12-08T20:22:16.744 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-12-08T20:22:17.114 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplejson. 2023-12-08T20:22:17.133 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../097-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-12-08T20:22:17.172 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-08T20:22:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package radosgw. 2023-12-08T20:22:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../098-radosgw_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking radosgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:18.871 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package rbd-fuse. 2023-12-08T20:22:18.887 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../099-rbd-fuse_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:18.928 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking rbd-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:19.265 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package rbd-mirror. 2023-12-08T20:22:19.282 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../100-rbd-mirror_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:19.348 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking rbd-mirror (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:19.905 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-08T20:22:19.923 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../082-libcephfs-dev_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:19.963 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking libcephfs-dev (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:19.975 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-attr (19.3.0-2) ... 2023-12-08T20:22:20.294 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-12-08T20:22:20.308 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package nvme-cli. 2023-12-08T20:22:20.325 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../083-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-12-08T20:22:20.384 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-12-08T20:22:20.420 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-12-08T20:22:20.555 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-12-08T20:22:20.740 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-atomicwrites. 2023-12-08T20:22:20.757 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../084-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-12-08T20:22:20.781 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:20.798 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-12-08T20:22:21.126 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-attr. 2023-12-08T20:22:21.148 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../085-python3-attr_19.3.0-2_all.deb ... 2023-12-08T20:22:21.184 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-12-08T20:22:21.504 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-webencodings. 2023-12-08T20:22:21.524 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../086-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-12-08T20:22:21.570 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-08T20:22:21.642 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user cephadm....done 2023-12-08T20:22:21.846 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-08T20:22:21.873 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-html5lib. 2023-12-08T20:22:21.890 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../087-python3-html5lib_1.0.1-2_all.deb ... 2023-12-08T20:22:21.931 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-12-08T20:22:22.231 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-08T20:22:22.310 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-12-08T20:22:22.326 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../088-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-12-08T20:22:22.368 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-08T20:22:22.524 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-12-08T20:22:22.650 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-12-08T20:22:22.846 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-packaging. 2023-12-08T20:22:22.867 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../089-python3-packaging_20.3-1_all.deb ... 2023-12-08T20:22:22.904 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-packaging (20.3-1) ... 2023-12-08T20:22:22.993 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-12-08T20:22:23.209 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pastescript. 2023-12-08T20:22:23.225 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../090-python3-pastescript_2.0.2-4_all.deb ... 2023-12-08T20:22:23.284 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-08T20:22:23.411 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-08T20:22:23.596 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pluggy. 2023-12-08T20:22:23.616 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../091-python3-pluggy_0.13.0-2_all.deb ... 2023-12-08T20:22:23.654 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-12-08T20:22:23.721 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-12-08T20:22:23.965 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-psutil. 2023-12-08T20:22:23.983 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../092-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-12-08T20:22:24.022 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-08T20:22:24.023 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-12-08T20:22:24.291 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-08T20:22:24.376 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-py. 2023-12-08T20:22:24.394 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../093-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-12-08T20:22:24.434 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-12-08T20:22:24.583 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-argparse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:24.763 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-08T20:22:24.781 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../094-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-12-08T20:22:24.821 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-08T20:22:24.918 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-08T20:22:25.044 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-12-08T20:22:25.107 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-08T20:22:25.127 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../095-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-12-08T20:22:25.165 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-08T20:22:25.353 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-08T20:22:25.493 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-pytest. 2023-12-08T20:22:25.512 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../096-python3-pytest_4.6.9-1_all.deb ... 2023-12-08T20:22:25.551 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-12-08T20:22:25.663 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-12-08T20:22:25.897 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package python3-simplejson. 2023-12-08T20:22:25.914 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../097-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-12-08T20:22:25.955 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-08T20:22:26.039 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-12-08T20:22:26.367 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-08T20:22:26.375 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package radosgw. 2023-12-08T20:22:26.393 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../098-radosgw_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:26.434 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking radosgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:26.745 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-12-08T20:22:27.063 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjaeger (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:27.188 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-12-08T20:22:27.556 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-12-08T20:22:27.612 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package rbd-fuse. 2023-12-08T20:22:27.632 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../099-rbd-fuse_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:27.670 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking rbd-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:27.866 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-packaging (20.3-1) ... 2023-12-08T20:22:28.048 INFO:teuthology.orchestra.run.smithi158.stdout:Selecting previously unselected package rbd-mirror. 2023-12-08T20:22:28.065 INFO:teuthology.orchestra.run.smithi158.stdout:Preparing to unpack .../100-rbd-mirror_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-08T20:22:28.106 INFO:teuthology.orchestra.run.smithi158.stdout:Unpacking rbd-mirror (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:28.175 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-12-08T20:22:28.301 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-08T20:22:28.585 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-12-08T20:22:28.724 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-attr (19.3.0-2) ... 2023-12-08T20:22:29.004 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-08T20:22:29.036 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-12-08T20:22:29.170 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-12-08T20:22:29.296 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-12-08T20:22:29.380 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-08T20:22:29.489 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:29.740 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up socat (1.7.3.3-2) ... 2023-12-08T20:22:29.868 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:30.102 INFO:teuthology.orchestra.run.smithi158.stdout:Adding system user cephadm....done 2023-12-08T20:22:30.202 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-12-08T20:22:30.264 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-08T20:22:30.545 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-08T20:22:30.623 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-08T20:22:30.679 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-12-08T20:22:30.780 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-12-08T20:22:30.908 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-12-08T20:22:31.025 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-12-08T20:22:31.069 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-08T20:22:31.377 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-12-08T20:22:31.386 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-08T20:22:31.539 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-12-08T20:22:31.772 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-08T20:22:31.841 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-12-08T20:22:32.065 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-12-08T20:22:32.167 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-12-08T20:22:32.349 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-08T20:22:32.460 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-08T20:22:32.587 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-08T20:22:32.634 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-08T20:22:32.898 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-12-08T20:22:32.943 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-ceph-argparse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:33.166 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-08T20:22:33.244 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-08T20:22:33.404 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-12-08T20:22:33.467 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-12-08T20:22:33.688 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-08T20:22:33.839 INFO:teuthology.orchestra.run.smithi069.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-12-08T20:22:33.975 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-12-08T20:22:34.111 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-12-08T20:22:34.237 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-08T20:22:34.353 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-12-08T20:22:34.588 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-12-08T20:22:34.673 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-08T20:22:34.940 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-08T20:22:35.015 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-12-08T20:22:35.074 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-08T20:22:35.200 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-12-08T20:22:35.300 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libjaeger (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:35.417 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-12-08T20:22:35.568 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-08T20:22:35.744 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-12-08T20:22:35.852 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librados2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:35.978 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-12-08T20:22:36.037 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-packaging (20.3-1) ... 2023-12-08T20:22:36.329 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-12-08T20:22:36.834 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-08T20:22:37.248 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-12-08T20:22:37.434 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librgw2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:37.560 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libsqlite3-mod-ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:37.633 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-08T20:22:37.719 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:37.845 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-12-08T20:22:37.986 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-08T20:22:38.222 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-12-08T20:22:38.338 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up socat (1.7.3.3-2) ... 2023-12-08T20:22:38.464 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:38.523 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-12-08T20:22:38.765 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-12-08T20:22:38.841 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-12-08T20:22:39.117 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-08T20:22:39.176 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:39.239 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-12-08T20:22:39.303 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-08T20:22:39.365 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-12-08T20:22:39.614 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libradosstriper1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:39.641 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-08T20:22:39.740 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-08T20:22:39.962 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-08T20:22:40.092 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-08T20:22:40.111 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-12-08T20:22:40.439 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-12-08T20:22:40.722 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-12-08T20:22:40.999 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-08T20:22:41.119 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-08T20:22:41.167 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librbd1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:41.293 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-08T20:22:41.419 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:41.419 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-12-08T20:22:41.570 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rsa (4.0-3) ... 2023-12-08T20:22:41.696 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-08T20:22:41.855 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-12-08T20:22:41.980 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-12-08T20:22:42.155 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-modules-core (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:42.282 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:42.405 INFO:teuthology.orchestra.run.smithi158.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-12-08T20:22:42.501 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-08T20:22:42.501 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-08T20:22:42.658 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-12-08T20:22:42.784 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-08T20:22:42.935 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs-dev (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:43.077 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-08T20:22:43.085 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-12-08T20:22:43.370 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-08T20:22:43.470 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-08T20:22:43.596 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-08T20:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-12-08T20:22:43.722 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-12-08T20:22:44.049 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-08T20:22:44.049 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rados (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:44.150 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rbd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:44.251 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up rbd-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:44.333 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up librados2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:44.369 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-routes (2.4.1-2) ... 2023-12-08T20:22:44.459 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-12-08T20:22:44.687 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-12-08T20:22:45.090 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:45.191 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cephfs (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:45.450 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-12-08T20:22:45.932 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up librgw2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:46.058 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libsqlite3-mod-ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:46.184 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:46.301 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-12-08T20:22:46.387 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:46.662 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-12-08T20:22:46.949 INFO:teuthology.orchestra.run.smithi069.stdout:Adding group ceph....done 2023-12-08T20:22:46.954 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-12-08T20:22:47.240 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-12-08T20:22:47.367 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user ceph....done 2023-12-08T20:22:47.543 INFO:teuthology.orchestra.run.smithi069.stdout:Setting system user ceph properties....done 2023-12-08T20:22:47.567 INFO:teuthology.orchestra.run.smithi069.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-08T20:22:47.585 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libcephfs2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:47.703 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-08T20:22:47.714 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-08T20:22:48.014 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-08T20:22:48.021 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libradosstriper1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:48.148 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-08T20:22:48.455 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-test (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:48.507 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-08T20:22:48.582 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up radosgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:48.826 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-08T20:22:48.826 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-08T20:22:49.458 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-base (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:49.575 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up librbd1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:49.701 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-08T20:22:49.827 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-12-08T20:22:49.843 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-08T20:22:49.953 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-rsa (4.0-3) ... 2023-12-08T20:22:50.254 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-12-08T20:22:50.277 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mds (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:50.504 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-08T20:22:50.505 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-08T20:22:50.538 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mgr-modules-core (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:50.667 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:50.894 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-08T20:22:50.894 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-08T20:22:50.928 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up rbd-mirror (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:51.134 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-08T20:22:51.135 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-08T20:22:51.335 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up libcephfs-dev (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:51.461 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-08T20:22:51.546 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:51.754 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-08T20:22:51.772 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-08T20:22:51.773 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-08T20:22:52.064 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-12-08T20:22:52.189 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-osd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:52.434 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-rados (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:52.559 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-rbd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:52.674 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-08T20:22:52.674 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-08T20:22:52.685 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up rbd-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:52.811 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-routes (2.4.1-2) ... 2023-12-08T20:22:53.121 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-12-08T20:22:53.133 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-k8sevents (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:53.259 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-diskprediction-local (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:53.401 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mon (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:53.483 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-rgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:53.609 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-cephfs (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:53.638 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-08T20:22:53.638 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-08T20:22:53.885 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-12-08T20:22:54.073 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:54.220 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:54.346 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-dashboard (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:54.490 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-08T20:22:54.789 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:54.791 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-08T20:22:55.301 INFO:teuthology.orchestra.run.smithi158.stdout:Adding group ceph....done 2023-12-08T20:22:55.678 INFO:teuthology.orchestra.run.smithi158.stdout:Adding system user ceph....done 2023-12-08T20:22:55.871 INFO:teuthology.orchestra.run.smithi158.stdout:Setting system user ceph properties....done 2023-12-08T20:22:55.896 INFO:teuthology.orchestra.run.smithi158.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-08T20:22:56.046 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-08T20:22:56.367 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-08T20:22:56.834 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-test (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:56.968 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up radosgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:57.215 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-08T20:22:57.215 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-08T20:22:57.637 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-08T20:22:57.862 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-base (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:58.086 INFO:teuthology.orchestra.run.smithi069.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-08T20:22:58.089 DEBUG:teuthology.parallel:result is None 2023-12-08T20:22:58.266 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-08T20:22:58.714 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mds (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:58.933 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-08T20:22:58.933 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-08T20:22:59.374 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up rbd-mirror (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:22:59.601 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-08T20:22:59.601 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-08T20:23:00.033 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mgr (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:00.264 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-08T20:23:00.264 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-08T20:23:00.701 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-osd (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:01.201 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-08T20:23:01.201 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-08T20:23:01.653 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mgr-k8sevents (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:01.779 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mgr-diskprediction-local (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:01.922 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mon (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:02.145 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-08T20:23:02.146 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-08T20:23:02.632 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mgr-cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:02.775 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:02.901 INFO:teuthology.orchestra.run.smithi158.stdout:Setting up ceph-mgr-dashboard (16.2.14-365-g451ceae1-1focal) ... 2023-12-08T20:23:03.045 INFO:teuthology.orchestra.run.smithi158.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-08T20:23:03.329 INFO:teuthology.orchestra.run.smithi158.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-08T20:23:06.174 INFO:teuthology.orchestra.run.smithi158.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-08T20:23:06.619 INFO:teuthology.orchestra.run.smithi158.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-08T20:23:06.622 DEBUG:teuthology.parallel:result is None 2023-12-08T20:23:06.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T20:23:06.766 DEBUG:teuthology.orchestra.run.smithi069:> dpkg-query -W -f '${Version}' ceph 2023-12-08T20:23:06.790 INFO:teuthology.orchestra.run.smithi069.stdout:16.2.14-365-g451ceae1-1focal 2023-12-08T20:23:06.795 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-365-g451ceae1-1focal 2023-12-08T20:23:06.795 INFO:teuthology.task.install:The correct ceph version 16.2.14-365-g451ceae1-1focal is installed. 2023-12-08T20:23:06.795 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T20:23:07.009 DEBUG:teuthology.orchestra.run.smithi158:> dpkg-query -W -f '${Version}' ceph 2023-12-08T20:23:07.034 INFO:teuthology.orchestra.run.smithi158.stdout:16.2.14-365-g451ceae1-1focal 2023-12-08T20:23:07.034 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-365-g451ceae1-1focal 2023-12-08T20:23:07.034 INFO:teuthology.task.install:The correct ceph version 16.2.14-365-g451ceae1-1focal is installed. 2023-12-08T20:23:07.035 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-08T20:23:07.035 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:07.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-08T20:23:07.051 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:07.051 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-08T20:23:07.068 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-08T20:23:07.068 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:07.068 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2023-12-08T20:23:07.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-08T20:23:07.118 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:07.118 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/daemon-helper 2023-12-08T20:23:07.134 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-08T20:23:07.149 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-08T20:23:07.150 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:07.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-08T20:23:07.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-08T20:23:07.222 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:07.222 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-08T20:23:07.238 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-08T20:23:07.248 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-08T20:23:07.248 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:07.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2023-12-08T20:23:07.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-08T20:23:07.323 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:07.323 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/stdin-killer 2023-12-08T20:23:07.337 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-08T20:23:07.350 INFO:teuthology.run_tasks:Running task ceph... 2023-12-08T20:23:07.420 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-08T20:23:07.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2023-12-08T20:23:07.423 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /var/log/ceph 2023-12-08T20:23:07.438 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-08T20:23:07.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-08T20:23:07.477 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-08T20:23:07.492 INFO:tasks.ceph:Creating extra log directories... 2023-12-08T20:23:07.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-08T20:23:07.494 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-08T20:23:07.565 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-08T20:23:07.565 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-08T20:23:07.565 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7485056', 'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7485056', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 8057, 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'targets': {'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi158.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-08_20:02:17', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-08T20:23:07.566 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-08T20:23:07.568 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-08T20:23:07.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2023-12-08T20:23:07.617 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/run/ceph 2023-12-08T20:23:07.634 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:07.634 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2023-12-08T20:23:07.638 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-08T20:23:07.638 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 361 Links: 1 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-08 20:19:34.846489241 +0000 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-08 20:19:34.546496145 +0000 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-08 20:19:34.546496145 +0000 2023-12-08T20:23:07.644 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-12-08T20:23:07.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-08T20:23:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-08T20:23:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-08T20:23:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000403728 s, 1.3 MB/s 2023-12-08T20:23:07.704 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-08T20:23:07.711 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2023-12-08T20:23:07.716 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-08T20:23:07.716 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:07.717 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 366 Links: 1 2023-12-08T20:23:07.717 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:07.717 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-08 20:19:35.102483349 +0000 2023-12-08T20:23:07.717 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-08 20:19:35.098483442 +0000 2023-12-08T20:23:07.717 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-08 20:19:35.098483442 +0000 2023-12-08T20:23:07.717 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-12-08T20:23:07.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-08T20:23:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-08T20:23:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-08T20:23:07.774 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000392575 s, 1.3 MB/s 2023-12-08T20:23:07.777 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-08T20:23:07.785 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2023-12-08T20:23:07.792 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-08T20:23:07.792 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:07.792 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 372 Links: 1 2023-12-08T20:23:07.792 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:07.792 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-08 20:19:35.606471750 +0000 2023-12-08T20:23:07.793 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-08 20:19:35.602471843 +0000 2023-12-08T20:23:07.793 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-08 20:19:35.602471843 +0000 2023-12-08T20:23:07.793 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-12-08T20:23:07.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-08T20:23:07.848 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-08T20:23:07.849 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-08T20:23:07.849 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000400241 s, 1.3 MB/s 2023-12-08T20:23:07.850 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-08T20:23:07.859 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2023-12-08T20:23:07.866 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-08T20:23:07.866 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:07.866 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-12-08T20:23:07.866 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:07.866 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-12-08 20:19:36.386453799 +0000 2023-12-08T20:23:07.866 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-12-08 20:19:36.102460335 +0000 2023-12-08T20:23:07.867 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-12-08 20:19:36.102460335 +0000 2023-12-08T20:23:07.867 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-12-08T20:23:07.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-08T20:23:07.879 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-12-08T20:23:07.879 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-12-08T20:23:07.879 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000329497 s, 1.6 MB/s 2023-12-08T20:23:07.880 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-08T20:23:07.932 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:07.932 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-08T20:23:07.933 INFO:tasks.ceph:Generating config... 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client] log to stderr = False 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:07.934 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:07.935 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:07.935 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-08T20:23:07.935 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-08T20:23:07.935 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-08T20:23:07.935 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-08T20:23:07.935 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-08T20:23:07.935 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-08T20:23:07.935 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-08T20:23:07.935 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-08T20:23:07.936 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-08T20:23:07.936 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-08T20:23:07.936 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-08T20:23:07.936 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-08T20:23:07.936 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-08T20:23:07.936 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-08T20:23:07.936 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-08T20:23:07.936 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-08T20:23:07.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-08T20:23:07.980 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/cluster1.keyring 2023-12-08T20:23:07.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-08T20:23:08.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-08T20:23:08.075 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.69')] 2023-12-08T20:23:08.076 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.69', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-08T20:23:08.077 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:08.077 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-08T20:23:08.081 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.69 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:08.154 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: generated fsid 17a8fb1d-0036-4b6c-a8d5-6296dd695374 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 0 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:fsid 17a8fb1d-0036-4b6c-a8d5-6296dd695374 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:last_changed 2023-12-08T20:23:08.154596+0000 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:created 2023-12-08T20:23:08.154596+0000 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:min_mon_release 0 (unknown) 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:election_strategy: 1 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2023-12-08T20:23:08.155 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-08T20:23:08.156 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-08T20:23:08.161 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 17a8fb1d-0036-4b6c-a8d5-6296dd695374... 2023-12-08T20:23:08.162 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-08T20:23:08.164 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-08T20:23:08.192 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-12-08T20:23:08.192 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-08T20:23:08.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2023-12-08T20:23:08.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2023-12-08T20:23:08.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = 172.21.15.69 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: mon client directed command retry = 5 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: ms inject socket failures = 5000 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 17a8fb1d-0036-4b6c-a8d5-6296dd695374 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.197 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = stupid 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2023-12-08T20:23:08.198 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:08.199 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:08.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2023-12-08T20:23:08.201 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2023-12-08T20:23:08.202 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-08T20:23:08.202 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-08T20:23:08.202 INFO:teuthology.orchestra.run.smithi069.stdout: debug rbd = 20 2023-12-08T20:23:08.202 INFO:teuthology.orchestra.run.smithi069.stdout: debug rbd_mirror = 15 2023-12-08T20:23:08.202 INFO:teuthology.orchestra.run.smithi069.stdout: log to stderr = False 2023-12-08T20:23:08.202 INFO:teuthology.orchestra.run.smithi069.stdout: rbd default features = 125 2023-12-08T20:23:08.202 INFO:teuthology.orchestra.run.smithi069.stdout: rbd mirror image policy type = none 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.0] 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.1] 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.2] 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.3] 2023-12-08T20:23:08.203 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.204 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.205 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2023-12-08T20:23:08.205 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2023-12-08T20:23:08.205 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-08T20:23:08.205 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2023-12-08T20:23:08.205 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.205 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2023-12-08T20:23:08.205 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-08T20:23:08.206 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2023-12-08T20:23:08.207 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2023-12-08T20:23:08.208 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = 172.21.15.69 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: mon client directed command retry = 5 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: ms inject socket failures = 5000 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = 17a8fb1d-0036-4b6c-a8d5-6296dd695374 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2023-12-08T20:23:08.209 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2023-12-08T20:23:08.210 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = stupid 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2023-12-08T20:23:08.211 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2023-12-08T20:23:08.212 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2023-12-08T20:23:08.213 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-08T20:23:08.214 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: debug rbd = 20 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: debug rbd_mirror = 15 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: log to stderr = False 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: rbd default features = 125 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: rbd mirror image policy type = none 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.0] 2023-12-08T20:23:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.1] 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.2] 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.3] 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:08.218 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-08T20:23:08.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-08T20:23:08.267 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-08T20:23:08.267 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:08.267 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-08T20:23:08.274 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:08.274 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-08T20:23:08.280 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2023-12-08T20:23:08.281 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:08.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-08T20:23:08.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-08T20:23:08.304 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:08.304 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:08.355 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi158.front.sepia.ceph.com 2023-12-08T20:23:08.355 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:08.355 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-08T20:23:08.356 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-08T20:23:08.382 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:08.383 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:08.387 INFO:tasks.ceph:Setting up mon nodes... 2023-12-08T20:23:08.388 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-08T20:23:08.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-08T20:23:08.432 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-08T20:23:08.434 INFO:tasks.ceph:Setting up mds nodes... 2023-12-08T20:23:08.435 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-08T20:23:08.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-08T20:23:08.465 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-08T20:23:08.476 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-08T20:23:08.531 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-08T20:23:08.545 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-08T20:23:08.576 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-08T20:23:08.588 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-08T20:23:08.618 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-08T20:23:08.630 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-08T20:23:08.667 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-08T20:23:08.680 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-08T20:23:08.709 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-08T20:23:08.722 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-08T20:23:08.800 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-08T20:23:08.814 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-08T20:23:08.845 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-08T20:23:08.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-08T20:23:08.889 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-08T20:23:08.901 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-08T20:23:08.902 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-08T20:23:08.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-08T20:23:08.919 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:08.919 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-08T20:23:08.919 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-08T20:23:08.919 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-08T20:23:09.439 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-08T20:23:09.439 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-08T20:23:09.439 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-08T20:23:09.439 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 2023-12-08T20:23:09.439 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-08T20:23:09.439 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-12-08T20:23:09.439 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-08T20:23:09.440 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-08T20:23:09.440 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-08T20:23:09.440 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-08T20:23:09.441 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-12-08T20:23:09.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-08T20:23:09.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-08T20:23:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:23:09.549 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-12-08T20:23:09.549 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-08T20:23:09.608 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:09.608 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-08T20:23:09.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-08T20:23:09.609 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-08T20:23:10.129 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-08T20:23:10.129 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-08T20:23:10.129 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-08T20:23:10.129 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 2023-12-08T20:23:10.129 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-08T20:23:10.129 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-12-08T20:23:10.129 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-08T20:23:10.130 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-08T20:23:10.130 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-08T20:23:10.130 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-08T20:23:10.132 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-12-08T20:23:10.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-08T20:23:10.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-08T20:23:10.174 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-12-08T20:23:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:23:10.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-08T20:23:10.232 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:10.232 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-08T20:23:10.232 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2023-12-08T20:23:10.232 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-08T20:23:10.705 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-08T20:23:10.706 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-08T20:23:10.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-12-08T20:23:10.707 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-08T20:23:10.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-08T20:23:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:23:10.772 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-12-08T20:23:10.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:10.895 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:10.889+0000 7f70413b7200 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-08T20:23:10.897 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:10.893+0000 7f70413b7200 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-08T20:23:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:10.893+0000 7f70413b7200 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-08T20:23:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:10.893+0000 7f70413b7200 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-08T20:23:10.898 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:10.893+0000 7f70413b7200 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-08T20:23:10.899 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:10.893+0000 7f70413b7200 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-08T20:23:13.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-08T20:23:13.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:13.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:13.933+0000 7fb68e792200 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-08T20:23:13.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:13.933+0000 7fb68e792200 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-08T20:23:13.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:13.933+0000 7fb68e792200 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-08T20:23:13.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:13.933+0000 7fb68e792200 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-08T20:23:13.937 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:13.933+0000 7fb68e792200 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-08T20:23:13.938 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:13.933+0000 7fb68e792200 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-08T20:23:16.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-08T20:23:16.976 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:17.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:17.004+0000 7f64d343f200 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-08T20:23:17.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:17.004+0000 7f64d343f200 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-08T20:23:17.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:17.004+0000 7f64d343f200 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-08T20:23:17.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:17.004+0000 7f64d343f200 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-08T20:23:17.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:17.004+0000 7f64d343f200 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-08T20:23:17.012 INFO:teuthology.orchestra.run.smithi069.stderr:2023-12-08T20:23:17.008+0000 7f64d343f200 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-08T20:23:19.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-08T20:23:20.012 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-08T20:23:20.013 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:20.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-08T20:23:20.071 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:20.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-08T20:23:20.081 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:20.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-08T20:23:20.094 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:20.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-08T20:23:20.105 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:20.105 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-08T20:23:20.112 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.112 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-08T20:23:20.120 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.120 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-08T20:23:20.127 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.127 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-08T20:23:20.132 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.133 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-08T20:23:20.138 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.138 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-08T20:23:20.144 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.144 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-08T20:23:20.196 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.197 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-08T20:23:20.203 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:20.203 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-08T20:23:20.210 INFO:tasks.ceph:Adding keys to all mons... 2023-12-08T20:23:20.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3Nl98DGGRAAOcsmLx4lxo1Tae1wX5/n7A== 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAue3NlDGlfNRAAB2s6b6v/HOVd0ytVobR9Jg== 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAxe3Nl1SbWNxAArWW+dDmeQhGCZNpkM182Gg== 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA1e3NliGitABAAbbn7za5lsSpiOqIUlVdm3g== 2023-12-08T20:23:20.225 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3NllPW4GxAAInuAHnr3HjGbZxZguCIxOQ== 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror] 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3NlI1OzHxAADkce+9pIgmuOyLN/tsh8fQ== 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.0] 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3Nl9xthIhAAE/M00LeXo1OSK84/F66YxA== 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.1] 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3NlwMLeJBAA8u35qYOvIpyRuyxiZJcIAQ== 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.2] 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3NlViXNJxAARqeQyHIoD6HYS3b1DYMKJg== 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.3] 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3NlV9dUKhAA7UWyRMJC/F1WL6iqmv+qsg== 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.4] 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3NlLU3CLxAAUwloXk9M/zRKImYskwQslA== 2023-12-08T20:23:20.226 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.5] 2023-12-08T20:23:20.227 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3Nl7w1uMhAAVyxXtYgV0mTamt1usdR7pg== 2023-12-08T20:23:20.227 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.6] 2023-12-08T20:23:20.227 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQAse3NljtQJNRAAbGq6Z3HCfyCrA5FBtBY5fg== 2023-12-08T20:23:20.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-08T20:23:20.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-08T20:23:20.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-08T20:23:20.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-08T20:23:20.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.559 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.819 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:20.907 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-08T20:23:20.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-08T20:23:20.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-08T20:23:20.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-08T20:23:20.990 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-08T20:23:21.038 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-08T20:23:21.038 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-08T20:23:21.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-08T20:23:21.041 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-08T20:23:21.041 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-08T20:23:21.041 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-08T20:23:21.041 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-08T20:23:21.043 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-08T20:23:21.043 INFO:tasks.ceph:Setting crush tunables to default 2023-12-08T20:23:21.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-08T20:23:21.267 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2023-12-08T20:23:21.277 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-08T20:23:21.277 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:21.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-08T20:23:21.291 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:21.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-08T20:23:21.346 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:21.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-08T20:23:21.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster cluster1 osd new 90b10638-13a2-4abb-aef1-bb5d60bbe5fa 0 2023-12-08T20:23:21.371 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:21.364+0000 7ff19502b500 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-08T20:23:21.432 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:21.428+0000 7ff19502b500 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-08T20:23:21.553 INFO:teuthology.orchestra.run.smithi069.stdout:0 2023-12-08T20:23:21.565 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster cluster1 osd new 8ecae547-9b2a-47dc-9e0f-759b25489aa9 1 2023-12-08T20:23:21.767 INFO:teuthology.orchestra.run.smithi069.stdout:1 2023-12-08T20:23:21.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster cluster1 osd new e4f82c05-9220-4901-b8ba-791d92d304b9 2 2023-12-08T20:23:21.973 INFO:teuthology.orchestra.run.smithi069.stdout:2 2023-12-08T20:23:21.989 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-08T20:23:21.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-08T20:23:21.991 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-08T20:23:21.992 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-08T20:23:21.992 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-08T20:23:21.994 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-08T20:23:21.994 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-08T20:23:21.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-08T20:23:21.996 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-08T20:23:21.996 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-08T20:23:22.101 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:22.096+0000 7ff19502b500 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-08T20:23:22.181 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:22.181 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"fsid":"17a8fb1d-0036-4b6c-a8d5-6296dd695374","created":"2023-12-08T20:23:21.155478+0000","modified":"2023-12-08T20:23:21.968738+0000","last_up_change":"0.000000","last_in_change":"2023-12-08T20:23:21.968738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"90b10638-13a2-4abb-aef1-bb5d60bbe5fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8ecae547-9b2a-47dc-9e0f-759b25489aa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e4f82c05-9220-4901-b8ba-791d92d304b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:22.192 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-08T20:23:22.192 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-08T20:23:22.342 INFO:tasks.ceph.cluster1.osd.0.smithi069.stderr:2023-12-08T20:23:22.336+0000 7feae1e2b200 -1 Falling back to public interface 2023-12-08T20:23:22.354 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:23:22.348+0000 7f2c7e205200 -1 Falling back to public interface 2023-12-08T20:23:22.358 INFO:tasks.ceph.cluster1.osd.1.smithi069.stderr:2023-12-08T20:23:22.352+0000 7f81a3611200 -1 Falling back to public interface 2023-12-08T20:23:22.495 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-08T20:23:22.662 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-08T20:23:22.663 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":5,"fsid":"17a8fb1d-0036-4b6c-a8d5-6296dd695374","created":"2023-12-08T20:23:21.155478+0000","modified":"2023-12-08T20:23:21.968738+0000","last_up_change":"0.000000","last_in_change":"2023-12-08T20:23:21.968738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"90b10638-13a2-4abb-aef1-bb5d60bbe5fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8ecae547-9b2a-47dc-9e0f-759b25489aa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e4f82c05-9220-4901-b8ba-791d92d304b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:22.672 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-08T20:23:22.682 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:22.676+0000 7ff19502b500 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-08T20:23:22.840 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:22.832+0000 7ff19502b500 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-08T20:23:23.072 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:23.064+0000 7ff19502b500 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-08T20:23:23.225 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:23.220+0000 7ff19502b500 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-08T20:23:23.273 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:23.268+0000 7ff19502b500 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-08T20:23:23.331 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:23.324+0000 7ff19502b500 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-08T20:23:23.383 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:23.376+0000 7ff19502b500 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-08T20:23:23.666 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:23.660+0000 7ff19502b500 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-08T20:23:23.820 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:23.812+0000 7ff19502b500 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-08T20:23:24.056 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.052+0000 7ff19502b500 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-08T20:23:24.121 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.116+0000 7ff19502b500 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-08T20:23:24.342 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.336+0000 7ff19502b500 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-08T20:23:24.398 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.392+0000 7ff19502b500 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-08T20:23:24.538 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.532+0000 7ff19502b500 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-08T20:23:24.629 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.624+0000 7ff19502b500 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-08T20:23:24.770 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.764+0000 7ff19502b500 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-08T20:23:24.953 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:24.948+0000 7ff19502b500 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-08T20:23:25.897 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:23:25.892+0000 7f2c7e205200 -1 osd.2 0 log_to_monitors {default=true} 2023-12-08T20:23:25.987 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T20:23:25.980+0000 7ff17ed32700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-08T20:23:26.071 INFO:tasks.ceph.cluster1.osd.0.smithi069.stderr:2023-12-08T20:23:26.064+0000 7feae1e2b200 -1 osd.0 0 log_to_monitors {default=true} 2023-12-08T20:23:26.145 INFO:tasks.ceph.cluster1.osd.1.smithi069.stderr:2023-12-08T20:23:26.140+0000 7f81a3611200 -1 osd.1 0 log_to_monitors {default=true} 2023-12-08T20:23:27.045 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:23:27.040+0000 7f2c799a5700 -1 osd.2 0 waiting for initial osdmap 2023-12-08T20:23:27.052 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:23:27.044+0000 7f2c76719700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-08T20:23:28.054 INFO:tasks.ceph.cluster1.osd.0.smithi069.stderr:2023-12-08T20:23:28.048+0000 7feadd5cb700 -1 osd.0 0 waiting for initial osdmap 2023-12-08T20:23:28.054 INFO:tasks.ceph.cluster1.osd.1.smithi069.stderr:2023-12-08T20:23:28.048+0000 7f819edb1700 -1 osd.1 0 waiting for initial osdmap 2023-12-08T20:23:28.059 INFO:tasks.ceph.cluster1.osd.0.smithi069.stderr:2023-12-08T20:23:28.052+0000 7feada33f700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-08T20:23:28.061 INFO:tasks.ceph.cluster1.osd.1.smithi069.stderr:2023-12-08T20:23:28.056+0000 7f819bb25700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-08T20:23:28.975 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-08T20:23:29.293 INFO:teuthology.misc.health.smithi069.stdout: 2023-12-08T20:23:29.293 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":9,"fsid":"17a8fb1d-0036-4b6c-a8d5-6296dd695374","created":"2023-12-08T20:23:21.155478+0000","modified":"2023-12-08T20:23:29.045376+0000","last_up_change":"2023-12-08T20:23:29.045376+0000","last_in_change":"2023-12-08T20:23:21.968738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"90b10638-13a2-4abb-aef1-bb5d60bbe5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":30922},{"type":"v1","addr":"172.21.15.69:6801","nonce":30922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":30922},{"type":"v1","addr":"172.21.15.69:6803","nonce":30922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":30922},{"type":"v1","addr":"172.21.15.69:6807","nonce":30922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":30922},{"type":"v1","addr":"172.21.15.69:6805","nonce":30922}]},"public_addr":"172.21.15.69:6801/30922","cluster_addr":"172.21.15.69:6803/30922","heartbeat_back_addr":"172.21.15.69:6807/30922","heartbeat_front_addr":"172.21.15.69:6805/30922","state":["exists","up"]},{"osd":1,"uuid":"8ecae547-9b2a-47dc-9e0f-759b25489aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":30923},{"type":"v1","addr":"172.21.15.69:6817","nonce":30923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":30923},{"type":"v1","addr":"172.21.15.69:6819","nonce":30923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":30923},{"type":"v1","addr":"172.21.15.69:6823","nonce":30923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":30923},{"type":"v1","addr":"172.21.15.69:6821","nonce":30923}]},"public_addr":"172.21.15.69:6817/30923","cluster_addr":"172.21.15.69:6819/30923","heartbeat_back_addr":"172.21.15.69:6823/30923","heartbeat_front_addr":"172.21.15.69:6821/30923","state":["exists","up"]},{"osd":2,"uuid":"e4f82c05-9220-4901-b8ba-791d92d304b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":30921},{"type":"v1","addr":"172.21.15.69:6809","nonce":30921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":30921},{"type":"v1","addr":"172.21.15.69:6811","nonce":30921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":30921},{"type":"v1","addr":"172.21.15.69:6815","nonce":30921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":30921},{"type":"v1","addr":"172.21.15.69:6813","nonce":30921}]},"public_addr":"172.21.15.69:6809/30921","cluster_addr":"172.21.15.69:6811/30921","heartbeat_back_addr":"172.21.15.69:6815/30921","heartbeat_front_addr":"172.21.15.69:6813/30921","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:26.932466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:29.305 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-08T20:23:29.305 INFO:tasks.ceph:Creating RBD pool 2023-12-08T20:23:29.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-08T20:23:30.077 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'rbd' created 2023-12-08T20:23:30.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-08T20:23:31.091 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-08T20:23:31.103 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-08T20:23:31.103 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-08T20:23:31.103 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-08T20:23:31.104 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-08T20:23:31.104 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-08T20:23:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:31.432 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":30735},{"type":"v1","addr":"172.21.15.69:6825","nonce":30735}]},"active_addr":"172.21.15.69:6825/30735","active_change":"2023-12-08T20:23:24.973948+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3963915111}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":721759237}]}]} 2023-12-08T20:23:31.434 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-08T20:23:31.434 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-08T20:23:31.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-08T20:23:31.702 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:31.703 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":11,"fsid":"17a8fb1d-0036-4b6c-a8d5-6296dd695374","created":"2023-12-08T20:23:21.155478+0000","modified":"2023-12-08T20:23:31.064909+0000","last_up_change":"2023-12-08T20:23:29.045376+0000","last_in_change":"2023-12-08T20:23:21.968738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-08T20:23:29.116791+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-08T20:23:29.644456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"90b10638-13a2-4abb-aef1-bb5d60bbe5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":30922},{"type":"v1","addr":"172.21.15.69:6801","nonce":30922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":30922},{"type":"v1","addr":"172.21.15.69:6803","nonce":30922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":30922},{"type":"v1","addr":"172.21.15.69:6807","nonce":30922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":30922},{"type":"v1","addr":"172.21.15.69:6805","nonce":30922}]},"public_addr":"172.21.15.69:6801/30922","cluster_addr":"172.21.15.69:6803/30922","heartbeat_back_addr":"172.21.15.69:6807/30922","heartbeat_front_addr":"172.21.15.69:6805/30922","state":["exists","up"]},{"osd":1,"uuid":"8ecae547-9b2a-47dc-9e0f-759b25489aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":30923},{"type":"v1","addr":"172.21.15.69:6817","nonce":30923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":30923},{"type":"v1","addr":"172.21.15.69:6819","nonce":30923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":30923},{"type":"v1","addr":"172.21.15.69:6823","nonce":30923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":30923},{"type":"v1","addr":"172.21.15.69:6821","nonce":30923}]},"public_addr":"172.21.15.69:6817/30923","cluster_addr":"172.21.15.69:6819/30923","heartbeat_back_addr":"172.21.15.69:6823/30923","heartbeat_front_addr":"172.21.15.69:6821/30923","state":["exists","up"]},{"osd":2,"uuid":"e4f82c05-9220-4901-b8ba-791d92d304b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":30921},{"type":"v1","addr":"172.21.15.69:6809","nonce":30921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":30921},{"type":"v1","addr":"172.21.15.69:6811","nonce":30921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":30921},{"type":"v1","addr":"172.21.15.69:6815","nonce":30921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":30921},{"type":"v1","addr":"172.21.15.69:6813","nonce":30921}]},"public_addr":"172.21.15.69:6809/30921","cluster_addr":"172.21.15.69:6811/30921","heartbeat_back_addr":"172.21.15.69:6815/30921","heartbeat_front_addr":"172.21.15.69:6813/30921","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:27.102958+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:27.160746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:26.932466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:31.716 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-08T20:23:31.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-08T20:23:31.988 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:31.988 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":11,"fsid":"17a8fb1d-0036-4b6c-a8d5-6296dd695374","created":"2023-12-08T20:23:21.155478+0000","modified":"2023-12-08T20:23:31.064909+0000","last_up_change":"2023-12-08T20:23:29.045376+0000","last_in_change":"2023-12-08T20:23:21.968738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-08T20:23:29.116791+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-08T20:23:29.644456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"90b10638-13a2-4abb-aef1-bb5d60bbe5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":30922},{"type":"v1","addr":"172.21.15.69:6801","nonce":30922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":30922},{"type":"v1","addr":"172.21.15.69:6803","nonce":30922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":30922},{"type":"v1","addr":"172.21.15.69:6807","nonce":30922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":30922},{"type":"v1","addr":"172.21.15.69:6805","nonce":30922}]},"public_addr":"172.21.15.69:6801/30922","cluster_addr":"172.21.15.69:6803/30922","heartbeat_back_addr":"172.21.15.69:6807/30922","heartbeat_front_addr":"172.21.15.69:6805/30922","state":["exists","up"]},{"osd":1,"uuid":"8ecae547-9b2a-47dc-9e0f-759b25489aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":30923},{"type":"v1","addr":"172.21.15.69:6817","nonce":30923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":30923},{"type":"v1","addr":"172.21.15.69:6819","nonce":30923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":30923},{"type":"v1","addr":"172.21.15.69:6823","nonce":30923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":30923},{"type":"v1","addr":"172.21.15.69:6821","nonce":30923}]},"public_addr":"172.21.15.69:6817/30923","cluster_addr":"172.21.15.69:6819/30923","heartbeat_back_addr":"172.21.15.69:6823/30923","heartbeat_front_addr":"172.21.15.69:6821/30923","state":["exists","up"]},{"osd":2,"uuid":"e4f82c05-9220-4901-b8ba-791d92d304b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":30921},{"type":"v1","addr":"172.21.15.69:6809","nonce":30921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":30921},{"type":"v1","addr":"172.21.15.69:6811","nonce":30921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":30921},{"type":"v1","addr":"172.21.15.69:6815","nonce":30921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":30921},{"type":"v1","addr":"172.21.15.69:6813","nonce":30921}]},"public_addr":"172.21.15.69:6809/30921","cluster_addr":"172.21.15.69:6811/30921","heartbeat_back_addr":"172.21.15.69:6815/30921","heartbeat_front_addr":"172.21.15.69:6813/30921","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:27.102958+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:27.160746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:26.932466+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:32.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-08T20:23:32.152 INFO:teuthology.orchestra.run.smithi069.stdout:38654705666 2023-12-08T20:23:32.152 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-08T20:23:32.298 INFO:teuthology.orchestra.run.smithi069.stdout:38654705667 2023-12-08T20:23:32.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-08T20:23:32.452 INFO:teuthology.orchestra.run.smithi069.stdout:34359738371 2023-12-08T20:23:32.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-08T20:23:32.738 INFO:teuthology.orchestra.run.smithi069.stdout:38654705665 2023-12-08T20:23:32.752 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705666 got 38654705665 for osd.0 2023-12-08T20:23:33.753 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-08T20:23:34.044 INFO:teuthology.orchestra.run.smithi069.stdout:38654705666 2023-12-08T20:23:34.057 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705666 got 38654705666 for osd.0 2023-12-08T20:23:34.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-08T20:23:34.328 INFO:teuthology.orchestra.run.smithi069.stdout:38654705667 2023-12-08T20:23:34.341 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705667 for osd.1 2023-12-08T20:23:34.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-08T20:23:34.620 INFO:teuthology.orchestra.run.smithi069.stdout:34359738371 2023-12-08T20:23:34.633 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2023-12-08T20:23:34.633 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-08T20:23:34.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-08T20:23:34.911 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:34.914 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-08T20:23:34.924 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-12-08T20:23:32.982774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223944,"statfs":{"total":289910292480,"available":288997318656,"internally_reserved":0,"allocated":294912,"data_stored":77531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.090003+0000","last_change":"2023-12-08T20:23:31.088552+0000","last_active":"2023-12-08T20:23:31.090003+0000","last_peered":"2023-12-08T20:23:31.090003+0000","last_clean":"2023-12-08T20:23:31.090003+0000","last_became_active":"2023-12-08T20:23:31.087873+0000","last_became_peered":"2023-12-08T20:23:31.087873+0000","last_unstale":"2023-12-08T20:23:31.090003+0000","last_undegraded":"2023-12-08T20:23:31.090003+0000","last_fullsized":"2023-12-08T20:23:31.090003+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.091944+0000","last_change":"2023-12-08T20:23:31.090536+0000","last_active":"2023-12-08T20:23:31.091944+0000","last_peered":"2023-12-08T20:23:31.091944+0000","last_clean":"2023-12-08T20:23:31.091944+0000","last_became_active":"2023-12-08T20:23:31.090232+0000","last_became_peered":"2023-12-08T20:23:31.090232+0000","last_unstale":"2023-12-08T20:23:31.091944+0000","last_undegraded":"2023-12-08T20:23:31.091944+0000","last_fullsized":"2023-12-08T20:23:31.091944+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.091988+0000","last_change":"2023-12-08T20:23:31.090543+0000","last_active":"2023-12-08T20:23:31.091988+0000","last_peered":"2023-12-08T20:23:31.091988+0000","last_clean":"2023-12-08T20:23:31.091988+0000","last_became_active":"2023-12-08T20:23:31.090237+0000","last_became_peered":"2023-12-08T20:23:31.090237+0000","last_unstale":"2023-12-08T20:23:31.091988+0000","last_undegraded":"2023-12-08T20:23:31.091988+0000","last_fullsized":"2023-12-08T20:23:31.091988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.092295+0000","last_change":"2023-12-08T20:23:31.091030+0000","last_active":"2023-12-08T20:23:31.092295+0000","last_peered":"2023-12-08T20:23:31.092295+0000","last_clean":"2023-12-08T20:23:31.092295+0000","last_became_active":"2023-12-08T20:23:31.090677+0000","last_became_peered":"2023-12-08T20:23:31.090677+0000","last_unstale":"2023-12-08T20:23:31.092295+0000","last_undegraded":"2023-12-08T20:23:31.092295+0000","last_fullsized":"2023-12-08T20:23:31.092295+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.086916+0000","last_change":"2023-12-08T20:23:31.086916+0000","last_active":"2023-12-08T20:23:31.086916+0000","last_peered":"2023-12-08T20:23:31.086916+0000","last_clean":"2023-12-08T20:23:31.086916+0000","last_became_active":"2023-12-08T20:23:31.086506+0000","last_became_peered":"2023-12-08T20:23:31.086506+0000","last_unstale":"2023-12-08T20:23:31.086916+0000","last_undegraded":"2023-12-08T20:23:31.086916+0000","last_fullsized":"2023-12-08T20:23:31.086916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.089525+0000","last_change":"2023-12-08T20:23:31.088313+0000","last_active":"2023-12-08T20:23:31.089525+0000","last_peered":"2023-12-08T20:23:31.089525+0000","last_clean":"2023-12-08T20:23:31.089525+0000","last_became_active":"2023-12-08T20:23:31.087708+0000","last_became_peered":"2023-12-08T20:23:31.087708+0000","last_unstale":"2023-12-08T20:23:31.089525+0000","last_undegraded":"2023-12-08T20:23:31.089525+0000","last_fullsized":"2023-12-08T20:23:31.089525+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.088915+0000","last_change":"2023-12-08T20:23:31.087602+0000","last_active":"2023-12-08T20:23:31.088915+0000","last_peered":"2023-12-08T20:23:31.088915+0000","last_clean":"2023-12-08T20:23:31.088915+0000","last_became_active":"2023-12-08T20:23:31.087202+0000","last_became_peered":"2023-12-08T20:23:31.087202+0000","last_unstale":"2023-12-08T20:23:31.088915+0000","last_undegraded":"2023-12-08T20:23:31.088915+0000","last_fullsized":"2023-12-08T20:23:31.088915+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.091260+0000","last_change":"2023-12-08T20:23:31.089393+0000","last_active":"2023-12-08T20:23:31.091260+0000","last_peered":"2023-12-08T20:23:31.091260+0000","last_clean":"2023-12-08T20:23:31.091260+0000","last_became_active":"2023-12-08T20:23:31.089161+0000","last_became_peered":"2023-12-08T20:23:31.089161+0000","last_unstale":"2023-12-08T20:23:31.091260+0000","last_undegraded":"2023-12-08T20:23:31.091260+0000","last_fullsized":"2023-12-08T20:23:31.091260+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.090044+0000","last_change":"2023-12-08T20:23:31.088728+0000","last_active":"2023-12-08T20:23:31.090044+0000","last_peered":"2023-12-08T20:23:31.090044+0000","last_clean":"2023-12-08T20:23:31.090044+0000","last_became_active":"2023-12-08T20:23:31.087836+0000","last_became_peered":"2023-12-08T20:23:31.087836+0000","last_unstale":"2023-12-08T20:23:31.090044+0000","last_undegraded":"2023-12-08T20:23:31.090044+0000","last_fullsized":"2023-12-08T20:23:31.090044+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297196,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074644,"statfs":{"total":96636764160,"available":96332435456,"internally_reserved":0,"allocated":102400,"data_stored":28095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297188,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074652,"statfs":{"total":96636764160,"available":96332443648,"internally_reserved":0,"allocated":94208,"data_stored":24183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":25253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T20:23:34.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-08T20:23:35.199 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:35.202 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2023-12-08T20:23:35.212 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-08T20:23:34.983308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891584,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223936,"statfs":{"total":289910292480,"available":288997310464,"internally_reserved":0,"allocated":303104,"data_stored":81443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.090003+0000","last_change":"2023-12-08T20:23:31.088552+0000","last_active":"2023-12-08T20:23:31.090003+0000","last_peered":"2023-12-08T20:23:31.090003+0000","last_clean":"2023-12-08T20:23:31.090003+0000","last_became_active":"2023-12-08T20:23:31.087873+0000","last_became_peered":"2023-12-08T20:23:31.087873+0000","last_unstale":"2023-12-08T20:23:31.090003+0000","last_undegraded":"2023-12-08T20:23:31.090003+0000","last_fullsized":"2023-12-08T20:23:31.090003+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.091944+0000","last_change":"2023-12-08T20:23:31.090536+0000","last_active":"2023-12-08T20:23:31.091944+0000","last_peered":"2023-12-08T20:23:31.091944+0000","last_clean":"2023-12-08T20:23:31.091944+0000","last_became_active":"2023-12-08T20:23:31.090232+0000","last_became_peered":"2023-12-08T20:23:31.090232+0000","last_unstale":"2023-12-08T20:23:31.091944+0000","last_undegraded":"2023-12-08T20:23:31.091944+0000","last_fullsized":"2023-12-08T20:23:31.091944+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.091988+0000","last_change":"2023-12-08T20:23:31.090543+0000","last_active":"2023-12-08T20:23:31.091988+0000","last_peered":"2023-12-08T20:23:31.091988+0000","last_clean":"2023-12-08T20:23:31.091988+0000","last_became_active":"2023-12-08T20:23:31.090237+0000","last_became_peered":"2023-12-08T20:23:31.090237+0000","last_unstale":"2023-12-08T20:23:31.091988+0000","last_undegraded":"2023-12-08T20:23:31.091988+0000","last_fullsized":"2023-12-08T20:23:31.091988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.092295+0000","last_change":"2023-12-08T20:23:31.091030+0000","last_active":"2023-12-08T20:23:31.092295+0000","last_peered":"2023-12-08T20:23:31.092295+0000","last_clean":"2023-12-08T20:23:31.092295+0000","last_became_active":"2023-12-08T20:23:31.090677+0000","last_became_peered":"2023-12-08T20:23:31.090677+0000","last_unstale":"2023-12-08T20:23:31.092295+0000","last_undegraded":"2023-12-08T20:23:31.092295+0000","last_fullsized":"2023-12-08T20:23:31.092295+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.086916+0000","last_change":"2023-12-08T20:23:31.086916+0000","last_active":"2023-12-08T20:23:31.086916+0000","last_peered":"2023-12-08T20:23:31.086916+0000","last_clean":"2023-12-08T20:23:31.086916+0000","last_became_active":"2023-12-08T20:23:31.086506+0000","last_became_peered":"2023-12-08T20:23:31.086506+0000","last_unstale":"2023-12-08T20:23:31.086916+0000","last_undegraded":"2023-12-08T20:23:31.086916+0000","last_fullsized":"2023-12-08T20:23:31.086916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.089525+0000","last_change":"2023-12-08T20:23:31.088313+0000","last_active":"2023-12-08T20:23:31.089525+0000","last_peered":"2023-12-08T20:23:31.089525+0000","last_clean":"2023-12-08T20:23:31.089525+0000","last_became_active":"2023-12-08T20:23:31.087708+0000","last_became_peered":"2023-12-08T20:23:31.087708+0000","last_unstale":"2023-12-08T20:23:31.089525+0000","last_undegraded":"2023-12-08T20:23:31.089525+0000","last_fullsized":"2023-12-08T20:23:31.089525+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.088915+0000","last_change":"2023-12-08T20:23:31.087602+0000","last_active":"2023-12-08T20:23:31.088915+0000","last_peered":"2023-12-08T20:23:31.088915+0000","last_clean":"2023-12-08T20:23:31.088915+0000","last_became_active":"2023-12-08T20:23:31.087202+0000","last_became_peered":"2023-12-08T20:23:31.087202+0000","last_unstale":"2023-12-08T20:23:31.088915+0000","last_undegraded":"2023-12-08T20:23:31.088915+0000","last_fullsized":"2023-12-08T20:23:31.088915+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.091260+0000","last_change":"2023-12-08T20:23:31.089393+0000","last_active":"2023-12-08T20:23:31.091260+0000","last_peered":"2023-12-08T20:23:31.091260+0000","last_clean":"2023-12-08T20:23:31.091260+0000","last_became_active":"2023-12-08T20:23:31.089161+0000","last_became_peered":"2023-12-08T20:23:31.089161+0000","last_unstale":"2023-12-08T20:23:31.091260+0000","last_undegraded":"2023-12-08T20:23:31.091260+0000","last_fullsized":"2023-12-08T20:23:31.091260+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-08T20:23:31.090044+0000","last_change":"2023-12-08T20:23:31.088728+0000","last_active":"2023-12-08T20:23:31.090044+0000","last_peered":"2023-12-08T20:23:31.090044+0000","last_clean":"2023-12-08T20:23:31.090044+0000","last_became_active":"2023-12-08T20:23:31.087836+0000","last_became_peered":"2023-12-08T20:23:31.087836+0000","last_unstale":"2023-12-08T20:23:31.090044+0000","last_undegraded":"2023-12-08T20:23:31.090044+0000","last_fullsized":"2023-12-08T20:23:31.090044+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:30.060232+0000","last_clean_scrub_stamp":"2023-12-08T20:23:30.060232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297196,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074644,"statfs":{"total":96636764160,"available":96332435456,"internally_reserved":0,"allocated":102400,"data_stored":28095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297188,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074652,"statfs":{"total":96636764160,"available":96332443648,"internally_reserved":0,"allocated":94208,"data_stored":24183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":106496,"data_stored":29165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T20:23:35.213 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-08T20:23:35.213 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-08T20:23:35.213 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-08T20:23:35.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-08T20:23:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:35.534 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-08T20:23:35.534 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-08T20:23:35.534 INFO:teuthology.run_tasks:Running task ceph... 2023-12-08T20:23:35.544 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-08T20:23:35.544 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-08T20:23:35.544 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7485056', 'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7485056', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-08_20:02:17-rbd-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 8057, 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'targets': {'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi158.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-08_20:02:17', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-08T20:23:35.545 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-08T20:23:35.547 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-08T20:23:35.556 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2023-12-08T20:23:35.558 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/run/ceph 2023-12-08T20:23:35.572 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:35.572 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2023-12-08T20:23:35.621 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-08T20:23:35.621 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_1 2023-12-08T20:23:35.628 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-08T20:23:35.628 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:35.628 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 346 Links: 1 2023-12-08T20:23:35.628 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:35.628 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-12-08 20:19:35.139013724 +0000 2023-12-08T20:23:35.628 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-12-08 20:19:34.839020638 +0000 2023-12-08T20:23:35.629 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-12-08 20:19:34.839020638 +0000 2023-12-08T20:23:35.629 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-12-08T20:23:35.629 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-08T20:23:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-12-08T20:23:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-12-08T20:23:35.640 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000306443 s, 1.7 MB/s 2023-12-08T20:23:35.641 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-08T20:23:35.691 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_2 2023-12-08T20:23:35.698 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-08T20:23:35.698 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:35.698 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 353 Links: 1 2023-12-08T20:23:35.698 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:35.698 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-12-08 20:19:35.343009021 +0000 2023-12-08T20:23:35.698 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-12-08 20:19:35.339009114 +0000 2023-12-08T20:23:35.699 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-12-08 20:19:35.339009114 +0000 2023-12-08T20:23:35.699 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-12-08T20:23:35.699 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-08T20:23:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-12-08T20:23:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-12-08T20:23:35.713 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000247132 s, 2.1 MB/s 2023-12-08T20:23:35.714 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-08T20:23:35.722 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_3 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 357 Links: 1 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-12-08 20:19:36.098991596 +0000 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-12-08 20:19:35.810998233 +0000 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-12-08 20:19:35.810998233 +0000 2023-12-08T20:23:35.773 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-12-08T20:23:35.774 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-08T20:23:35.788 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-12-08T20:23:35.789 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-12-08T20:23:35.789 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.00028832 s, 1.8 MB/s 2023-12-08T20:23:35.790 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-08T20:23:35.799 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_4 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-12-08 20:19:36.614979701 +0000 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-12-08 20:19:36.314986616 +0000 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-12-08 20:19:36.314986616 +0000 2023-12-08T20:23:35.849 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-12-08T20:23:35.850 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-08T20:23:35.862 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-12-08T20:23:35.863 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-12-08T20:23:35.863 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000293511 s, 1.7 MB/s 2023-12-08T20:23:35.864 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-08T20:23:35.873 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:35.873 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-08T20:23:35.873 INFO:tasks.ceph:Generating config... 2023-12-08T20:23:35.874 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-08T20:23:35.874 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-08T20:23:35.874 INFO:tasks.ceph:[client] log to stderr = False 2023-12-08T20:23:35.874 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-08T20:23:35.874 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-08T20:23:35.874 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:35.874 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:35.875 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:35.875 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:35.875 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:35.875 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:35.875 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:35.875 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:35.875 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-08T20:23:35.875 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-08T20:23:35.875 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-08T20:23:35.875 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-08T20:23:35.875 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-08T20:23:35.875 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-08T20:23:35.875 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-08T20:23:35.875 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-08T20:23:35.875 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-08T20:23:35.876 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-08T20:23:35.877 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-08T20:23:35.877 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-08T20:23:35.877 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-08T20:23:35.948 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.keyring 2023-12-08T20:23:35.951 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-08T20:23:35.983 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-08T20:23:35.993 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.158')] 2023-12-08T20:23:35.993 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.158', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-08T20:23:35.994 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:35.994 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-08T20:23:36.000 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.158 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:36.029 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:36.029 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: generated fsid 2c73137c-b639-4d21-81b1-3b2ddefe3289 2023-12-08T20:23:36.029 INFO:teuthology.orchestra.run.smithi158.stdout:epoch 0 2023-12-08T20:23:36.029 INFO:teuthology.orchestra.run.smithi158.stdout:fsid 2c73137c-b639-4d21-81b1-3b2ddefe3289 2023-12-08T20:23:36.029 INFO:teuthology.orchestra.run.smithi158.stdout:last_changed 2023-12-08T20:23:36.029385+0000 2023-12-08T20:23:36.030 INFO:teuthology.orchestra.run.smithi158.stdout:created 2023-12-08T20:23:36.029385+0000 2023-12-08T20:23:36.030 INFO:teuthology.orchestra.run.smithi158.stdout:min_mon_release 0 (unknown) 2023-12-08T20:23:36.030 INFO:teuthology.orchestra.run.smithi158.stdout:election_strategy: 1 2023-12-08T20:23:36.030 INFO:teuthology.orchestra.run.smithi158.stdout:0: [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon.a 2023-12-08T20:23:36.030 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-08T20:23:36.031 DEBUG:teuthology.orchestra.run.smithi158:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-08T20:23:36.035 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 2c73137c-b639-4d21-81b1-3b2ddefe3289... 2023-12-08T20:23:36.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-08T20:23:36.038 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-08T20:23:36.081 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-12-08T20:23:36.081 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2023-12-08T20:23:36.081 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-08T20:23:36.081 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2023-12-08T20:23:36.081 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-08T20:23:36.082 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2023-12-08T20:23:36.083 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.084 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = 172.21.15.158 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: mon client directed command retry = 5 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: ms inject socket failures = 5000 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 2c73137c-b639-4d21-81b1-3b2ddefe3289 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2023-12-08T20:23:36.085 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.086 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = stupid 2023-12-08T20:23:36.087 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2023-12-08T20:23:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-12-08T20:23:36.089 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.090 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: debug rbd = 20 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: debug rbd_mirror = 15 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: log to stderr = False 2023-12-08T20:23:36.091 INFO:teuthology.orchestra.run.smithi069.stdout: rbd default features = 125 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout: rbd mirror image policy type = none 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.0] 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.1] 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.2] 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.092 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.093 INFO:teuthology.orchestra.run.smithi069.stdout:[client.mirror.3] 2023-12-08T20:23:36.093 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.093 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.115 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2023-12-08T20:23:36.116 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2023-12-08T20:23:36.117 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = 172.21.15.158 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: mon client directed command retry = 5 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: ms inject socket failures = 5000 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = 2c73137c-b639-4d21-81b1-3b2ddefe3289 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2023-12-08T20:23:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2023-12-08T20:23:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = stupid 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2023-12-08T20:23:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2023-12-08T20:23:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-12-08T20:23:36.122 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.123 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-08T20:23:36.124 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout: debug rbd = 20 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout: debug rbd_mirror = 15 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout: log to stderr = False 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout: rbd default features = 125 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout: rbd mirror image policy type = none 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.0] 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.125 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.1] 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.2] 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.3] 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-08T20:23:36.126 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-08T20:23:36.127 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-08T20:23:36.127 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-08T20:23:36.174 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-08T20:23:36.174 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:36.174 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-08T20:23:36.181 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:36.182 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-08T20:23:36.228 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2023-12-08T20:23:36.229 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:36.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-08T20:23:36.229 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-08T20:23:36.255 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:36.255 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:36.260 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi158.front.sepia.ceph.com 2023-12-08T20:23:36.260 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:36.260 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-08T20:23:36.260 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-08T20:23:36.288 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:36.288 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:36.336 INFO:tasks.ceph:Setting up mon nodes... 2023-12-08T20:23:36.337 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-08T20:23:36.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-08T20:23:36.382 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-08T20:23:36.384 INFO:tasks.ceph:Setting up mds nodes... 2023-12-08T20:23:36.384 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-08T20:23:36.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-08T20:23:36.464 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-08T20:23:36.477 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-08T20:23:36.514 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-08T20:23:36.530 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-08T20:23:36.560 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-08T20:23:36.572 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-08T20:23:36.648 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-08T20:23:36.661 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-08T20:23:36.693 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-08T20:23:36.704 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-08T20:23:36.737 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-08T20:23:36.750 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-08T20:23:36.779 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-08T20:23:36.791 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-08T20:23:36.870 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-08T20:23:36.884 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-08T20:23:36.918 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-08T20:23:36.930 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-08T20:23:36.931 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-08T20:23:36.931 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-08T20:23:36.943 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:36.943 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-08T20:23:36.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com 2023-12-08T20:23:36.943 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-08T20:23:37.029 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-08T20:23:37.029 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-08T20:23:37.030 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-08T20:23:37.031 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-12-08T20:23:37.032 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-08T20:23:37.052 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-08T20:23:37.058 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: /sbin/restorecon: command not found 2023-12-08T20:23:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:23:37.059 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-08T20:23:37.119 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:37.119 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-08T20:23:37.119 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com 2023-12-08T20:23:37.119 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-08T20:23:37.160 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-08T20:23:37.160 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-08T20:23:37.161 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-08T20:23:37.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-12-08T20:23:37.162 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-08T20:23:37.183 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-08T20:23:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:23:37.196 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: /sbin/restorecon: command not found 2023-12-08T20:23:37.196 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-08T20:23:37.206 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-08T20:23:37.206 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-08T20:23:37.206 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com 2023-12-08T20:23:37.206 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-08T20:23:37.247 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-08T20:23:37.248 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-08T20:23:37.248 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-08T20:23:37.248 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-12-08T20:23:37.249 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-08T20:23:37.310 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-08T20:23:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:23:37.319 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: /sbin/restorecon: command not found 2023-12-08T20:23:37.319 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:37.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:37.387+0000 7f12cb61b200 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-08T20:23:37.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:37.387+0000 7f12cb61b200 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-08T20:23:37.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:37.387+0000 7f12cb61b200 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-08T20:23:37.391 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:37.387+0000 7f12cb61b200 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-08T20:23:37.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:37.387+0000 7f12cb61b200 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-08T20:23:37.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:37.387+0000 7f12cb61b200 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-08T20:23:40.358 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-08T20:23:40.371 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:40.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:40.451+0000 7fdee19b9200 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-08T20:23:40.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:40.451+0000 7fdee19b9200 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-08T20:23:40.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:40.451+0000 7fdee19b9200 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-08T20:23:40.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:40.451+0000 7fdee19b9200 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-08T20:23:40.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:40.451+0000 7fdee19b9200 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-08T20:23:40.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:40.451+0000 7fdee19b9200 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-08T20:23:43.419 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-08T20:23:43.435 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:43.467+0000 7f16effbc200 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-08T20:23:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:43.467+0000 7f16effbc200 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-08T20:23:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:43.471+0000 7f16effbc200 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-08T20:23:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:43.471+0000 7f16effbc200 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-08T20:23:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:43.471+0000 7f16effbc200 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-08T20:23:43.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-12-08T20:23:43.471+0000 7f16effbc200 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-08T20:23:46.458 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-08T20:23:46.519 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-08T20:23:46.520 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:46.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-08T20:23:46.536 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.536 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-08T20:23:46.550 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.550 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-08T20:23:46.561 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.561 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-08T20:23:46.615 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-12-08T20:23:46.615 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-08T20:23:46.622 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.622 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-08T20:23:46.628 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.628 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-08T20:23:46.635 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.635 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-08T20:23:46.685 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.685 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-08T20:23:46.692 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.692 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-08T20:23:46.698 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.698 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-08T20:23:46.748 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.749 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-08T20:23:46.757 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:46.757 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-08T20:23:46.761 INFO:tasks.ceph:Adding keys to all mons... 2023-12-08T20:23:46.761 DEBUG:teuthology.orchestra.run.smithi158:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-08T20:23:46.776 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr.x] 2023-12-08T20:23:46.776 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3Nllx2+FhAAoKn+oPcS9E1+RLd0vgzoqQ== 2023-12-08T20:23:46.777 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.0] 2023-12-08T20:23:46.777 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBJe3Nl93BNFxAADRCxPhBoO9UBNiNeiH0giw== 2023-12-08T20:23:46.777 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.1] 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBMe3NlJT02GxAAv/b6hl1agYXdvLNEgishmw== 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.2] 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBPe3Nl9hZHHBAAUIy5KD5Yq7kbDXVLik1A8w== 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3Nll/mjGxAARfBvvudXd3g5izz83ZzdxA== 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror] 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3NlNhupHhAAR2V0bzE1XBmumbLzk6F0ug== 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.0] 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3Nlz4NqIRAAieD3r1LyWN3qNzQo3UUJqw== 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.1] 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3NlE+2qJhAAiHNDshAxJxzRihltmQB8Mg== 2023-12-08T20:23:46.778 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.2] 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3NlxulWKRAAoxddj3k+4kw176axx3ejzA== 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.3] 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3NluCXzKxAAV4x75V5knLT6LWVPWhIlag== 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.4] 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3NllMp4LhAASzyGfNwjOmi8uEb5kGWJZw== 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.5] 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3NliaTeMxAA8Rh3uFCdbQGrX50LR5kdkw== 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout:[client.mirror.6] 2023-12-08T20:23:46.779 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBIe3NlecK5NhAAhDC1Pjvw3nFNYs2rl6gYJg== 2023-12-08T20:23:46.780 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-08T20:23:46.817 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-08T20:23:46.904 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-08T20:23:46.944 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-08T20:23:46.984 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.025 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.112 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.155 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.198 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.282 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.325 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.364 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.452 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-08T20:23:47.499 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-08T20:23:47.499 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-08T20:23:47.514 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-08T20:23:47.581 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-08T20:23:47.596 DEBUG:teuthology.orchestra.run.smithi158:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-08T20:23:47.602 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-08T20:23:47.603 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-08T20:23:47.603 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-08T20:23:47.605 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-08T20:23:47.605 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-08T20:23:47.605 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-08T20:23:47.606 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-08T20:23:47.608 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-08T20:23:47.609 INFO:tasks.ceph:Setting crush tunables to default 2023-12-08T20:23:47.609 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-08T20:23:47.819 INFO:teuthology.orchestra.run.smithi158.stderr:adjusted tunables profile to default 2023-12-08T20:23:47.830 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-08T20:23:47.830 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:47.830 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-08T20:23:47.846 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:47.846 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-08T20:23:47.859 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:23:47.859 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-08T20:23:47.868 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster cluster2 osd new 062430ae-7a0d-4126-a49a-fd9272a96c6c 0 2023-12-08T20:23:47.985 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:47.980+0000 7fb531509500 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-08T20:23:48.035 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:48.028+0000 7fb531509500 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-08T20:23:48.115 INFO:teuthology.orchestra.run.smithi158.stdout:0 2023-12-08T20:23:48.129 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster cluster2 osd new 663f2371-8213-45da-8045-6daa41e524a0 1 2023-12-08T20:23:48.319 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:48.312+0000 7fb531509500 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-08T20:23:48.325 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-12-08T20:23:48.462 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster cluster2 osd new 42b2ab24-f2ac-4168-bb0f-c864ce7f0c09 2 2023-12-08T20:23:48.676 INFO:teuthology.orchestra.run.smithi158.stdout:2 2023-12-08T20:23:48.690 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-08T20:23:48.690 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-08T20:23:48.693 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-08T20:23:48.693 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-08T20:23:48.693 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-08T20:23:48.696 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-08T20:23:48.696 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-08T20:23:48.697 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-08T20:23:48.699 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-08T20:23:48.700 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-08T20:23:48.751 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:48.744+0000 7fb531509500 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-08T20:23:48.865 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:48.865 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":5,"fsid":"2c73137c-b639-4d21-81b1-3b2ddefe3289","created":"2023-12-08T20:23:47.719036+0000","modified":"2023-12-08T20:23:48.671599+0000","last_up_change":"0.000000","last_in_change":"2023-12-08T20:23:48.671599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"062430ae-7a0d-4126-a49a-fd9272a96c6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"663f2371-8213-45da-8045-6daa41e524a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"42b2ab24-f2ac-4168-bb0f-c864ce7f0c09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:48.877 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-08T20:23:48.878 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-08T20:23:48.895 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:48.888+0000 7fb531509500 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-08T20:23:49.089 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.088+0000 7fb531509500 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-08T20:23:49.180 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-08T20:23:49.241 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.236+0000 7fb531509500 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-08T20:23:49.288 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.284+0000 7fb531509500 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-08T20:23:49.308 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:23:49.303+0000 7fb5420f1200 -1 Falling back to public interface 2023-12-08T20:23:49.332 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:23:49.327+0000 7f670e9fb200 -1 Falling back to public interface 2023-12-08T20:23:49.332 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:23:49.327+0000 7f5887da8200 -1 Falling back to public interface 2023-12-08T20:23:49.346 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.344+0000 7fb531509500 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-08T20:23:49.361 INFO:teuthology.misc.health.smithi158.stdout: 2023-12-08T20:23:49.361 INFO:teuthology.misc.health.smithi158.stdout:{"epoch":5,"fsid":"2c73137c-b639-4d21-81b1-3b2ddefe3289","created":"2023-12-08T20:23:47.719036+0000","modified":"2023-12-08T20:23:48.671599+0000","last_up_change":"0.000000","last_in_change":"2023-12-08T20:23:48.671599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"062430ae-7a0d-4126-a49a-fd9272a96c6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"663f2371-8213-45da-8045-6daa41e524a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"42b2ab24-f2ac-4168-bb0f-c864ce7f0c09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:49.371 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-08T20:23:49.395 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.392+0000 7fb531509500 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-08T20:23:49.667 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.664+0000 7fb531509500 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-08T20:23:49.724 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.720+0000 7fb531509500 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-08T20:23:49.953 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:49.948+0000 7fb531509500 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-08T20:23:50.115 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:50.112+0000 7fb531509500 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-08T20:23:50.225 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:50.224+0000 7fb531509500 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-08T20:23:50.278 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:50.276+0000 7fb531509500 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-08T20:23:50.415 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:50.412+0000 7fb531509500 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-08T20:23:50.504 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:50.500+0000 7fb531509500 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-08T20:23:50.733 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:50.732+0000 7fb531509500 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-08T20:23:50.786 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:50.784+0000 7fb531509500 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-08T20:23:51.821 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:23:51.816+0000 7fb51b327700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-08T20:23:52.997 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:23:52.991+0000 7fb5420f1200 -1 osd.0 0 log_to_monitors {default=true} 2023-12-08T20:23:53.082 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:23:53.075+0000 7f5887da8200 -1 osd.2 0 log_to_monitors {default=true} 2023-12-08T20:23:53.153 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:23:53.147+0000 7f670e9fb200 -1 osd.1 0 log_to_monitors {default=true} 2023-12-08T20:23:54.714 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:23:54.707+0000 7fb53d891700 -1 osd.0 0 waiting for initial osdmap 2023-12-08T20:23:54.717 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:23:54.711+0000 7f5883548700 -1 osd.2 0 waiting for initial osdmap 2023-12-08T20:23:54.717 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:23:54.711+0000 7f670b9c0700 -1 osd.1 0 waiting for initial osdmap 2023-12-08T20:23:54.719 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:23:54.715+0000 7fb53a605700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-08T20:23:54.723 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:23:54.719+0000 7f6706f0f700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-08T20:23:54.724 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:23:54.719+0000 7f58802bc700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-08T20:23:55.675 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-08T20:23:55.943 INFO:teuthology.misc.health.smithi158.stdout: 2023-12-08T20:23:55.943 INFO:teuthology.misc.health.smithi158.stdout:{"epoch":8,"fsid":"2c73137c-b639-4d21-81b1-3b2ddefe3289","created":"2023-12-08T20:23:47.719036+0000","modified":"2023-12-08T20:23:55.706485+0000","last_up_change":"2023-12-08T20:23:55.706485+0000","last_in_change":"2023-12-08T20:23:48.671599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"062430ae-7a0d-4126-a49a-fd9272a96c6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":31044},{"type":"v1","addr":"172.21.15.158:6801","nonce":31044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":31044},{"type":"v1","addr":"172.21.15.158:6803","nonce":31044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":31044},{"type":"v1","addr":"172.21.15.158:6807","nonce":31044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":31044},{"type":"v1","addr":"172.21.15.158:6805","nonce":31044}]},"public_addr":"172.21.15.158:6801/31044","cluster_addr":"172.21.15.158:6803/31044","heartbeat_back_addr":"172.21.15.158:6807/31044","heartbeat_front_addr":"172.21.15.158:6805/31044","state":["exists","up"]},{"osd":1,"uuid":"663f2371-8213-45da-8045-6daa41e524a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":31052},{"type":"v1","addr":"172.21.15.158:6809","nonce":31052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6811","nonce":31052},{"type":"v1","addr":"172.21.15.158:6813","nonce":31052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":31052},{"type":"v1","addr":"172.21.15.158:6822","nonce":31052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6815","nonce":31052},{"type":"v1","addr":"172.21.15.158:6817","nonce":31052}]},"public_addr":"172.21.15.158:6809/31052","cluster_addr":"172.21.15.158:6813/31052","heartbeat_back_addr":"172.21.15.158:6822/31052","heartbeat_front_addr":"172.21.15.158:6817/31052","state":["exists","up"]},{"osd":2,"uuid":"42b2ab24-f2ac-4168-bb0f-c864ce7f0c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":31060},{"type":"v1","addr":"172.21.15.158:6812","nonce":31060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":31060},{"type":"v1","addr":"172.21.15.158:6816","nonce":31060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":31060},{"type":"v1","addr":"172.21.15.158:6823","nonce":31060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":31060},{"type":"v1","addr":"172.21.15.158:6819","nonce":31060}]},"public_addr":"172.21.15.158:6812/31060","cluster_addr":"172.21.15.158:6816/31060","heartbeat_back_addr":"172.21.15.158:6823/31060","heartbeat_front_addr":"172.21.15.158:6819/31060","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:55.955 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-08T20:23:55.955 INFO:tasks.ceph:Creating RBD pool 2023-12-08T20:23:55.955 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-08T20:23:56.733 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'rbd' created 2023-12-08T20:23:56.744 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-08T20:23:57.748 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-08T20:23:57.759 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-08T20:23:57.760 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-08T20:23:57.760 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-08T20:23:57.760 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-08T20:23:57.760 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-08T20:23:58.071 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:58.084 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":34014},{"type":"v1","addr":"172.21.15.69:6827","nonce":34014}]},"active_addr":"172.21.15.69:6827/34014","active_change":"2023-12-08T20:23:50.806956+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":4175544156}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2145198756}]}]} 2023-12-08T20:23:58.086 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-08T20:23:58.086 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-08T20:23:58.086 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-08T20:23:58.366 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:58.366 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":10,"fsid":"2c73137c-b639-4d21-81b1-3b2ddefe3289","created":"2023-12-08T20:23:47.719036+0000","modified":"2023-12-08T20:23:57.727829+0000","last_up_change":"2023-12-08T20:23:55.706485+0000","last_in_change":"2023-12-08T20:23:48.671599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-08T20:23:55.774683+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-08T20:23:56.295214+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"062430ae-7a0d-4126-a49a-fd9272a96c6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":31044},{"type":"v1","addr":"172.21.15.158:6801","nonce":31044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":31044},{"type":"v1","addr":"172.21.15.158:6803","nonce":31044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":31044},{"type":"v1","addr":"172.21.15.158:6807","nonce":31044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":31044},{"type":"v1","addr":"172.21.15.158:6805","nonce":31044}]},"public_addr":"172.21.15.158:6801/31044","cluster_addr":"172.21.15.158:6803/31044","heartbeat_back_addr":"172.21.15.158:6807/31044","heartbeat_front_addr":"172.21.15.158:6805/31044","state":["exists","up"]},{"osd":1,"uuid":"663f2371-8213-45da-8045-6daa41e524a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":31052},{"type":"v1","addr":"172.21.15.158:6809","nonce":31052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6811","nonce":31052},{"type":"v1","addr":"172.21.15.158:6813","nonce":31052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":31052},{"type":"v1","addr":"172.21.15.158:6822","nonce":31052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6815","nonce":31052},{"type":"v1","addr":"172.21.15.158:6817","nonce":31052}]},"public_addr":"172.21.15.158:6809/31052","cluster_addr":"172.21.15.158:6813/31052","heartbeat_back_addr":"172.21.15.158:6822/31052","heartbeat_front_addr":"172.21.15.158:6817/31052","state":["exists","up"]},{"osd":2,"uuid":"42b2ab24-f2ac-4168-bb0f-c864ce7f0c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":31060},{"type":"v1","addr":"172.21.15.158:6812","nonce":31060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":31060},{"type":"v1","addr":"172.21.15.158:6816","nonce":31060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":31060},{"type":"v1","addr":"172.21.15.158:6823","nonce":31060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":31060},{"type":"v1","addr":"172.21.15.158:6819","nonce":31060}]},"public_addr":"172.21.15.158:6812/31060","cluster_addr":"172.21.15.158:6816/31060","heartbeat_back_addr":"172.21.15.158:6823/31060","heartbeat_front_addr":"172.21.15.158:6819/31060","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:53.995758+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:54.105390+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:54.062187+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:58.379 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-08T20:23:58.380 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-08T20:23:58.653 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:23:58.656 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":10,"fsid":"2c73137c-b639-4d21-81b1-3b2ddefe3289","created":"2023-12-08T20:23:47.719036+0000","modified":"2023-12-08T20:23:57.727829+0000","last_up_change":"2023-12-08T20:23:55.706485+0000","last_in_change":"2023-12-08T20:23:48.671599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-08T20:23:55.774683+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-08T20:23:56.295214+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"062430ae-7a0d-4126-a49a-fd9272a96c6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":31044},{"type":"v1","addr":"172.21.15.158:6801","nonce":31044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":31044},{"type":"v1","addr":"172.21.15.158:6803","nonce":31044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":31044},{"type":"v1","addr":"172.21.15.158:6807","nonce":31044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":31044},{"type":"v1","addr":"172.21.15.158:6805","nonce":31044}]},"public_addr":"172.21.15.158:6801/31044","cluster_addr":"172.21.15.158:6803/31044","heartbeat_back_addr":"172.21.15.158:6807/31044","heartbeat_front_addr":"172.21.15.158:6805/31044","state":["exists","up"]},{"osd":1,"uuid":"663f2371-8213-45da-8045-6daa41e524a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":31052},{"type":"v1","addr":"172.21.15.158:6809","nonce":31052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6811","nonce":31052},{"type":"v1","addr":"172.21.15.158:6813","nonce":31052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":31052},{"type":"v1","addr":"172.21.15.158:6822","nonce":31052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6815","nonce":31052},{"type":"v1","addr":"172.21.15.158:6817","nonce":31052}]},"public_addr":"172.21.15.158:6809/31052","cluster_addr":"172.21.15.158:6813/31052","heartbeat_back_addr":"172.21.15.158:6822/31052","heartbeat_front_addr":"172.21.15.158:6817/31052","state":["exists","up"]},{"osd":2,"uuid":"42b2ab24-f2ac-4168-bb0f-c864ce7f0c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":31060},{"type":"v1","addr":"172.21.15.158:6812","nonce":31060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":31060},{"type":"v1","addr":"172.21.15.158:6816","nonce":31060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":31060},{"type":"v1","addr":"172.21.15.158:6823","nonce":31060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":31060},{"type":"v1","addr":"172.21.15.158:6819","nonce":31060}]},"public_addr":"172.21.15.158:6812/31060","cluster_addr":"172.21.15.158:6816/31060","heartbeat_back_addr":"172.21.15.158:6823/31060","heartbeat_front_addr":"172.21.15.158:6819/31060","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:53.995758+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:54.105390+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-08T20:23:54.062187+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-08T20:23:58.667 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-08T20:23:58.814 INFO:teuthology.orchestra.run.smithi158.stdout:34359738370 2023-12-08T20:23:58.815 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-08T20:23:58.961 INFO:teuthology.orchestra.run.smithi158.stdout:34359738370 2023-12-08T20:23:58.962 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-08T20:23:59.111 INFO:teuthology.orchestra.run.smithi158.stdout:34359738371 2023-12-08T20:23:59.111 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-08T20:23:59.391 INFO:teuthology.orchestra.run.smithi158.stdout:34359738369 2023-12-08T20:23:59.412 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738370 got 34359738369 for osd.0 2023-12-08T20:24:00.413 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-08T20:24:00.699 INFO:teuthology.orchestra.run.smithi158.stdout:34359738370 2023-12-08T20:24:00.713 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738370 got 34359738370 for osd.0 2023-12-08T20:24:00.713 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-08T20:24:00.990 INFO:teuthology.orchestra.run.smithi158.stdout:34359738369 2023-12-08T20:24:01.002 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738370 got 34359738369 for osd.1 2023-12-08T20:24:02.004 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-08T20:24:02.288 INFO:teuthology.orchestra.run.smithi158.stdout:34359738371 2023-12-08T20:24:02.302 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738370 got 34359738371 for osd.1 2023-12-08T20:24:02.302 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-08T20:24:02.573 INFO:teuthology.orchestra.run.smithi158.stdout:34359738372 2023-12-08T20:24:02.585 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738372 for osd.2 2023-12-08T20:24:02.586 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-08T20:24:02.586 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-08T20:24:02.865 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:24:02.868 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-12-08T20:24:02.879 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-08T20:24:02.818006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223944,"statfs":{"total":289910292480,"available":288997318656,"internally_reserved":0,"allocated":294912,"data_stored":80424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748826+0000","last_change":"2023-12-08T20:23:57.746929+0000","last_active":"2023-12-08T20:23:57.748826+0000","last_peered":"2023-12-08T20:23:57.748826+0000","last_clean":"2023-12-08T20:23:57.748826+0000","last_became_active":"2023-12-08T20:23:57.746567+0000","last_became_peered":"2023-12-08T20:23:57.746567+0000","last_unstale":"2023-12-08T20:23:57.748826+0000","last_undegraded":"2023-12-08T20:23:57.748826+0000","last_fullsized":"2023-12-08T20:23:57.748826+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748219+0000","last_change":"2023-12-08T20:23:57.744164+0000","last_active":"2023-12-08T20:23:57.748219+0000","last_peered":"2023-12-08T20:23:57.748219+0000","last_clean":"2023-12-08T20:23:57.748219+0000","last_became_active":"2023-12-08T20:23:57.743454+0000","last_became_peered":"2023-12-08T20:23:57.743454+0000","last_unstale":"2023-12-08T20:23:57.748219+0000","last_undegraded":"2023-12-08T20:23:57.748219+0000","last_fullsized":"2023-12-08T20:23:57.748219+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748290+0000","last_change":"2023-12-08T20:23:57.744704+0000","last_active":"2023-12-08T20:23:57.748290+0000","last_peered":"2023-12-08T20:23:57.748290+0000","last_clean":"2023-12-08T20:23:57.748290+0000","last_became_active":"2023-12-08T20:23:57.744361+0000","last_became_peered":"2023-12-08T20:23:57.744361+0000","last_unstale":"2023-12-08T20:23:57.748290+0000","last_undegraded":"2023-12-08T20:23:57.748290+0000","last_fullsized":"2023-12-08T20:23:57.748290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.747127+0000","last_change":"2023-12-08T20:23:57.743842+0000","last_active":"2023-12-08T20:23:57.747127+0000","last_peered":"2023-12-08T20:23:57.747127+0000","last_clean":"2023-12-08T20:23:57.747127+0000","last_became_active":"2023-12-08T20:23:57.743531+0000","last_became_peered":"2023-12-08T20:23:57.743531+0000","last_unstale":"2023-12-08T20:23:57.747127+0000","last_undegraded":"2023-12-08T20:23:57.747127+0000","last_fullsized":"2023-12-08T20:23:57.747127+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.747015+0000","last_change":"2023-12-08T20:23:57.745552+0000","last_active":"2023-12-08T20:23:57.747015+0000","last_peered":"2023-12-08T20:23:57.747015+0000","last_clean":"2023-12-08T20:23:57.747015+0000","last_became_active":"2023-12-08T20:23:57.745134+0000","last_became_peered":"2023-12-08T20:23:57.745134+0000","last_unstale":"2023-12-08T20:23:57.747015+0000","last_undegraded":"2023-12-08T20:23:57.747015+0000","last_fullsized":"2023-12-08T20:23:57.747015+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748626+0000","last_change":"2023-12-08T20:23:57.747874+0000","last_active":"2023-12-08T20:23:57.748626+0000","last_peered":"2023-12-08T20:23:57.748626+0000","last_clean":"2023-12-08T20:23:57.748626+0000","last_became_active":"2023-12-08T20:23:57.747540+0000","last_became_peered":"2023-12-08T20:23:57.747540+0000","last_unstale":"2023-12-08T20:23:57.748626+0000","last_undegraded":"2023-12-08T20:23:57.748626+0000","last_fullsized":"2023-12-08T20:23:57.748626+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.746810+0000","last_change":"2023-12-08T20:23:57.746059+0000","last_active":"2023-12-08T20:23:57.746810+0000","last_peered":"2023-12-08T20:23:57.746810+0000","last_clean":"2023-12-08T20:23:57.746810+0000","last_became_active":"2023-12-08T20:23:57.745567+0000","last_became_peered":"2023-12-08T20:23:57.745567+0000","last_unstale":"2023-12-08T20:23:57.746810+0000","last_undegraded":"2023-12-08T20:23:57.746810+0000","last_fullsized":"2023-12-08T20:23:57.746810+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748339+0000","last_change":"2023-12-08T20:23:57.744870+0000","last_active":"2023-12-08T20:23:57.748339+0000","last_peered":"2023-12-08T20:23:57.748339+0000","last_clean":"2023-12-08T20:23:57.748339+0000","last_became_active":"2023-12-08T20:23:57.744576+0000","last_became_peered":"2023-12-08T20:23:57.744576+0000","last_unstale":"2023-12-08T20:23:57.748339+0000","last_undegraded":"2023-12-08T20:23:57.748339+0000","last_fullsized":"2023-12-08T20:23:57.748339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748349+0000","last_change":"2023-12-08T20:23:57.745158+0000","last_active":"2023-12-08T20:23:57.748349+0000","last_peered":"2023-12-08T20:23:57.748349+0000","last_clean":"2023-12-08T20:23:57.748349+0000","last_became_active":"2023-12-08T20:23:57.744579+0000","last_became_peered":"2023-12-08T20:23:57.744579+0000","last_unstale":"2023-12-08T20:23:57.748349+0000","last_undegraded":"2023-12-08T20:23:57.748349+0000","last_fullsized":"2023-12-08T20:23:57.748349+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T20:24:02.879 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-08T20:24:03.145 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:24:03.147 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-12-08T20:24:03.158 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-08T20:24:02.818006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223944,"statfs":{"total":289910292480,"available":288997318656,"internally_reserved":0,"allocated":294912,"data_stored":80424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748826+0000","last_change":"2023-12-08T20:23:57.746929+0000","last_active":"2023-12-08T20:23:57.748826+0000","last_peered":"2023-12-08T20:23:57.748826+0000","last_clean":"2023-12-08T20:23:57.748826+0000","last_became_active":"2023-12-08T20:23:57.746567+0000","last_became_peered":"2023-12-08T20:23:57.746567+0000","last_unstale":"2023-12-08T20:23:57.748826+0000","last_undegraded":"2023-12-08T20:23:57.748826+0000","last_fullsized":"2023-12-08T20:23:57.748826+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748219+0000","last_change":"2023-12-08T20:23:57.744164+0000","last_active":"2023-12-08T20:23:57.748219+0000","last_peered":"2023-12-08T20:23:57.748219+0000","last_clean":"2023-12-08T20:23:57.748219+0000","last_became_active":"2023-12-08T20:23:57.743454+0000","last_became_peered":"2023-12-08T20:23:57.743454+0000","last_unstale":"2023-12-08T20:23:57.748219+0000","last_undegraded":"2023-12-08T20:23:57.748219+0000","last_fullsized":"2023-12-08T20:23:57.748219+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748290+0000","last_change":"2023-12-08T20:23:57.744704+0000","last_active":"2023-12-08T20:23:57.748290+0000","last_peered":"2023-12-08T20:23:57.748290+0000","last_clean":"2023-12-08T20:23:57.748290+0000","last_became_active":"2023-12-08T20:23:57.744361+0000","last_became_peered":"2023-12-08T20:23:57.744361+0000","last_unstale":"2023-12-08T20:23:57.748290+0000","last_undegraded":"2023-12-08T20:23:57.748290+0000","last_fullsized":"2023-12-08T20:23:57.748290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.747127+0000","last_change":"2023-12-08T20:23:57.743842+0000","last_active":"2023-12-08T20:23:57.747127+0000","last_peered":"2023-12-08T20:23:57.747127+0000","last_clean":"2023-12-08T20:23:57.747127+0000","last_became_active":"2023-12-08T20:23:57.743531+0000","last_became_peered":"2023-12-08T20:23:57.743531+0000","last_unstale":"2023-12-08T20:23:57.747127+0000","last_undegraded":"2023-12-08T20:23:57.747127+0000","last_fullsized":"2023-12-08T20:23:57.747127+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.747015+0000","last_change":"2023-12-08T20:23:57.745552+0000","last_active":"2023-12-08T20:23:57.747015+0000","last_peered":"2023-12-08T20:23:57.747015+0000","last_clean":"2023-12-08T20:23:57.747015+0000","last_became_active":"2023-12-08T20:23:57.745134+0000","last_became_peered":"2023-12-08T20:23:57.745134+0000","last_unstale":"2023-12-08T20:23:57.747015+0000","last_undegraded":"2023-12-08T20:23:57.747015+0000","last_fullsized":"2023-12-08T20:23:57.747015+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748626+0000","last_change":"2023-12-08T20:23:57.747874+0000","last_active":"2023-12-08T20:23:57.748626+0000","last_peered":"2023-12-08T20:23:57.748626+0000","last_clean":"2023-12-08T20:23:57.748626+0000","last_became_active":"2023-12-08T20:23:57.747540+0000","last_became_peered":"2023-12-08T20:23:57.747540+0000","last_unstale":"2023-12-08T20:23:57.748626+0000","last_undegraded":"2023-12-08T20:23:57.748626+0000","last_fullsized":"2023-12-08T20:23:57.748626+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.746810+0000","last_change":"2023-12-08T20:23:57.746059+0000","last_active":"2023-12-08T20:23:57.746810+0000","last_peered":"2023-12-08T20:23:57.746810+0000","last_clean":"2023-12-08T20:23:57.746810+0000","last_became_active":"2023-12-08T20:23:57.745567+0000","last_became_peered":"2023-12-08T20:23:57.745567+0000","last_unstale":"2023-12-08T20:23:57.746810+0000","last_undegraded":"2023-12-08T20:23:57.746810+0000","last_fullsized":"2023-12-08T20:23:57.746810+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748339+0000","last_change":"2023-12-08T20:23:57.744870+0000","last_active":"2023-12-08T20:23:57.748339+0000","last_peered":"2023-12-08T20:23:57.748339+0000","last_clean":"2023-12-08T20:23:57.748339+0000","last_became_active":"2023-12-08T20:23:57.744576+0000","last_became_peered":"2023-12-08T20:23:57.744576+0000","last_unstale":"2023-12-08T20:23:57.748339+0000","last_undegraded":"2023-12-08T20:23:57.748339+0000","last_fullsized":"2023-12-08T20:23:57.748339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-08T20:23:57.748349+0000","last_change":"2023-12-08T20:23:57.745158+0000","last_active":"2023-12-08T20:23:57.748349+0000","last_peered":"2023-12-08T20:23:57.748349+0000","last_clean":"2023-12-08T20:23:57.748349+0000","last_became_active":"2023-12-08T20:23:57.744579+0000","last_became_peered":"2023-12-08T20:23:57.744579+0000","last_unstale":"2023-12-08T20:23:57.748349+0000","last_undegraded":"2023-12-08T20:23:57.748349+0000","last_fullsized":"2023-12-08T20:23:57.748349+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-08T20:23:56.715698+0000","last_clean_scrub_stamp":"2023-12-08T20:23:56.715698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-08T20:24:03.159 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-08T20:24:03.159 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-08T20:24:03.159 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-08T20:24:03.159 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-08T20:24:03.466 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-12-08T20:24:03.479 INFO:teuthology.orchestra.run.smithi158.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-08T20:24:03.480 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-08T20:24:03.480 INFO:teuthology.run_tasks:Running task exec... 2023-12-08T20:24:03.490 INFO:teuthology.task.exec:Executing custom commands... 2023-12-08T20:24:03.490 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi158.front.sepia.ceph.com 2023-12-08T20:24:03.490 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:03.846 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror 2023-12-08T20:24:03.859 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:04.215 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.0 2023-12-08T20:24:04.228 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:04.591 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.1 2023-12-08T20:24:04.604 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:04.957 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.2 2023-12-08T20:24:04.969 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:05.325 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.3 2023-12-08T20:24:05.338 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:05.686 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror 2023-12-08T20:24:05.703 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:06.055 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.0 2023-12-08T20:24:06.067 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:06.424 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.1 2023-12-08T20:24:06.436 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:06.790 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.2 2023-12-08T20:24:06.800 DEBUG:teuthology.orchestra.run.smithi158:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-08T20:24:07.146 INFO:teuthology.orchestra.run.smithi158.stderr:updated caps for client.mirror.3 2023-12-08T20:24:07.157 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-08T20:24:07.169 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-08T20:24:07.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-08T20:24:07.171 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-08T20:24:07.185 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-08T20:24:07.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-08T20:24:07.187 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-08T20:24:07.197 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-08T20:24:07.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-08T20:24:07.199 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-08T20:24:07.211 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-08T20:24:07.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-08T20:24:07.212 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-08T20:24:07.223 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-08T20:24:07.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-08T20:24:07.225 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-08T20:24:07.236 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-08T20:24:07.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-08T20:24:07.238 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-08T20:24:07.249 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-08T20:24:07.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-08T20:24:07.251 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-08T20:24:07.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-08T20:24:07.262 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-08T20:24:07.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-08T20:24:07.264 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-08T20:24:07.276 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702067047 2023-12-08T20:24:07.277 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-08T20:24:07.277 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-08T20:24:07.300 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702067047 2023-12-08T20:24:07.300 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-08T20:24:07.300 INFO:teuthology.run_tasks:Running task workunit... 2023-12-08T20:24:07.315 INFO:tasks.workunit:Pulling workunits from ref 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T20:24:07.316 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-08T20:24:07.316 DEBUG:teuthology.orchestra.run.smithi158:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-08T20:24:07.317 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-08T20:24:07.317 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-08T20:24:07.317 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-08T20:24:07.317 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2023-12-08T20:24:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:24:07.326 INFO:teuthology.orchestra.run.smithi158.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-08T20:24:07.326 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-08T20:24:07.376 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-08T20:24:07.376 DEBUG:teuthology.orchestra.run.smithi158:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-08T20:24:07.382 INFO:tasks.workunit:timeout=6h 2023-12-08T20:24:07.383 INFO:tasks.workunit:cleanup=True 2023-12-08T20:24:07.383 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T20:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-08T20:25:08.319 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6563552365417407) > weight (0.25) 2023-12-08T20:25:08.319 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3586316912921891) > weight (0.25) 2023-12-08T20:25:08.319 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8790786446128458) > weight (0.25) 2023-12-08T20:25:08.319 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-08T20:25:08.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-08T20:25:08.321 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-08T20:25:08.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:25:08.315+0000 7fcb08856700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 34300) UID: 1000 2023-12-08T20:25:08.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:25:09.322 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-08T20:25:09.322 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:25:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:25:09.322 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-08T20:25:09.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-08T20:25:09.323 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-08T20:25:09.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-08T20:25:09.328 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2023-12-08T20:25:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Note: switching to '451ceae11b5d4caee05e94372fe658621eefc0b1'. 2023-12-08T20:25:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-08T20:25:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-08T20:25:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:state without impacting any branches by switching back to a branch. 2023-12-08T20:25:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-08T20:25:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-08T20:25:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: git switch -c 2023-12-08T20:25:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Or undo this operation with: 2023-12-08T20:25:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: git switch - 2023-12-08T20:25:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-08T20:25:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:HEAD is now at 451ceae11b5 Merge branch 'wip-63745-pacific' of https://github.com/baergj/ceph into wip-yuri5-testing-2023-12-07-0740-pacific 2023-12-08T20:25:11.755 DEBUG:teuthology.orchestra.run.smithi158:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-08T20:25:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-08T20:25:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-08T20:25:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-08T20:25:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-08T20:25:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-08T20:25:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-08T20:25:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-08T20:25:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-08T20:25:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-08T20:25:12.853 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-12-08T20:25:12.853 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-08T20:25:12.901 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-08T20:25:12.902 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-08T20:25:12.903 DEBUG:teuthology.orchestra.run.smithi158:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=451ceae11b5d4caee05e94372fe658621eefc0b1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-08T20:25:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ IMAGE_COUNT=50 2023-12-08T20:25:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ export LOCKDEP=0 2023-12-08T20:25:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-08T20:25:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-08T20:25:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ type xmlstarlet 2023-12-08T20:25:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ XMLSTARLET=xmlstarlet 2023-12-08T20:25:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ RBD_MIRROR_INSTANCES=4 2023-12-08T20:25:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CLUSTER1=cluster1 2023-12-08T20:25:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CLUSTER2=cluster2 2023-12-08T20:25:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ PEER_CLUSTER_SUFFIX= 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ POOL=mirror 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ PARENT_POOL=mirror_parent 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ NS1=ns1 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ NS2=ns2 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ TEMPDIR= 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ID=mirror 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ MIRROR_USER_ID_PREFIX=mirror. 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ MIRROR_POOL_MODE=image 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ MIRROR_IMAGE_MODE=snapshot 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ export CEPH_ARGS=--id mirror 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ LAST_MIRROR_INSTANCE=3 2023-12-08T20:25:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-08T20:25:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-08T20:25:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-08T20:25:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_BIN=. 2023-12-08T20:25:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_SRC=. 2023-12-08T20:25:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -e CMakeCache.txt ] 2023-12-08T20:25:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 -gt 0 ] 2023-12-08T20:25:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ setup 2023-12-08T20:25:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local c 2023-12-08T20:25:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ trap cleanup $? INT TERM EXIT 2023-12-08T20:25:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ setup_tempdir 2023-12-08T20:25:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mktemp -d 2023-12-08T20:25:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ TEMPDIR=/tmp/tmp.b8SaxlVk4h 2023-12-08T20:25:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -z 1 ] 2023-12-08T20:25:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ setup_pools cluster1 cluster2 2023-12-08T20:25:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local remote_cluster=cluster2 2023-12-08T20:25:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mon_map_file 2023-12-08T20:25:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mon_addr 2023-12-08T20:25:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local admin_key_file 2023-12-08T20:25:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local uuid 2023-12-08T20:25:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-08T20:25:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:pool 'mirror' already exists 2023-12-08T20:25:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-08T20:25:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:pool 'mirror_parent' already exists 2023-12-08T20:25:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror 2023-12-08T20:25:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror_parent 2023-12-08T20:25:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-08T20:25:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-08T20:25:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-08T20:25:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-08T20:25:21.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-08T20:25:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-08T20:25:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -z ] 2023-12-08T20:25:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -z ] 2023-12-08T20:25:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_add cluster1 mirror cluster2 2023-12-08T20:25:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local client_cluster=cluster2 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local remote_cluster=cluster2 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local uuid_var_name 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local error_code 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local peer_uuid 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:25:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-08T20:25:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_uuid=b843ce86-9440-45ca-8974-40c42e366514 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=0 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 -eq 17 ] 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 0 -eq 0 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror_parent 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local client_cluster=cluster2 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local remote_cluster=cluster2 2023-12-08T20:25:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local uuid_var_name 2023-12-08T20:25:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local error_code 2023-12-08T20:25:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local peer_uuid 2023-12-08T20:25:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:25:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_uuid=11d951e7-3456-4114-ba07-859714a29881 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=0 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 -eq 17 ] 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 0 -eq 0 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ setup_pools cluster2 cluster1 2023-12-08T20:25:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local remote_cluster=cluster1 2023-12-08T20:25:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mon_map_file 2023-12-08T20:25:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mon_addr 2023-12-08T20:25:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local admin_key_file 2023-12-08T20:25:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local uuid 2023-12-08T20:25:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-08T20:25:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:pool 'mirror' already exists 2023-12-08T20:25:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-08T20:25:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:pool 'mirror_parent' already exists 2023-12-08T20:25:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror 2023-12-08T20:25:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror_parent 2023-12-08T20:25:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-08T20:25:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-08T20:25:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-08T20:25:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-08T20:25:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-08T20:25:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-08T20:25:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -z ] 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -z ] 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_add cluster2 mirror cluster1 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local client_cluster=cluster1 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local remote_cluster=cluster1 2023-12-08T20:25:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local uuid_var_name 2023-12-08T20:25:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local error_code 2023-12-08T20:25:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local peer_uuid 2023-12-08T20:25:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:25:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-08T20:25:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_uuid=ae4b71a7-a32c-42a9-8e83-59fa08bcd1ab 2023-12-08T20:25:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=0 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 -eq 17 ] 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 0 -eq 0 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror_parent 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local client_cluster=cluster1 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local remote_cluster=cluster1 2023-12-08T20:25:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local uuid_var_name 2023-12-08T20:25:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local error_code 2023-12-08T20:25:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local peer_uuid 2023-12-08T20:25:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:25:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-08T20:25:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ peer_uuid=9cdc907b-3adf-41aa-980a-7341bf93c73f 2023-12-08T20:25:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=0 2023-12-08T20:25:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:25:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 -eq 17 ] 2023-12-08T20:25:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 0 -eq 0 2023-12-08T20:25:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:25:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:25:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirrors cluster1 2023-12-08T20:25:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:25:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster1:0 2023-12-08T20:25:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:25:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:25:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster1:1 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster1:2 2023-12-08T20:25:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:2 2023-12-08T20:25:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-08T20:25:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:2 2023-12-08T20:25:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster1:2 ] 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster1:3 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:3 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:3 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster1:3 ] 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirrors cluster2 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster2:0 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2:0 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster2:0 2023-12-08T20:25:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster2:0 ] 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster2 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster2:1 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2:1 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-08T20:25:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster2:1 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster2:1 ] 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster2 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster2:2 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2:2 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster2:2 2023-12-08T20:25:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster2:2 ] 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster2 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ start_mirror cluster2:3 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2:3 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster2:3 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster2:3 ] 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster2 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster2 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 1 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ testlog TEST: add image and test replay after client crashes 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + teedate -a +%F %T /tmp/tmp.b8SaxlVk4h/rbd-mirror.test.log 2023-12-08T20:25:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo 2023-12-08 20:25:30 TEST: add image and test replay after client crashes 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:2023-12-08 20:25:30 TEST: add image and test replay after client crashes 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=test 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror test 512M 2023-12-08T20:25:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 512M ] 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=512M 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2023-12-08T20:25:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:25:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-08T20:25:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:25:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-08T20:25:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:25:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109a7cbb7cf2 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109a7cbb7cf2 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:25:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:30 2023 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:25:30 2023 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:30 2023 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:25:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:25:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:25:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:25:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:25:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:25:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:25:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:25:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + sleep 0 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:25:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:25:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:25:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:25:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:25:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:25:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:25:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:25:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:25:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:25:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:2 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:2 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster1:2 ] 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:2 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:2 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster1:2 ] 2023-12-08T20:25:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-08T20:25:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-08T20:25:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-08T20:25:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:3 2023-12-08T20:25:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:3 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster1:3 ] 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:3 2023-12-08T20:25:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:3 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster1:3 ] 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-08T20:25:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-08T20:25:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:25:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:25:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:25:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:25:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:25:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:2 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:2 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster1:2 ] 2023-12-08T20:25:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:2 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:2 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster1:2 ] 2023-12-08T20:25:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-08T20:25:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-08T20:25:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:3 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:3 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster1:3 ] 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:3 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:3 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster1:3 ] 2023-12-08T20:25:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-08T20:25:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-08T20:25:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 2 2023-12-08T20:25:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:25:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:25:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:25:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:25:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:25:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:25:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:25:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:25:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:25:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:25:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:25:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:2 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:2 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster1:2 ] 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:2 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:2 2023-12-08T20:25:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 2 = cluster1:2 ] 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=2 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=2 2023-12-08T20:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-08T20:25:36.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-08T20:25:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-08T20:25:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:3 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:3 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster1:3 ] 2023-12-08T20:25:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:3 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:3 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 3 = cluster1:3 ] 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=3 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=3 2023-12-08T20:25:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-08T20:25:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-08T20:25:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-08T20:25:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 4 2023-12-08T20:25:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:25:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:25:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:25:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:25:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:25:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:25:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:25:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:25:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:25:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:25:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:25:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:25:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:25:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:25:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:25:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:25:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:25:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:25:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:25:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:25:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:25:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:25:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:25:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:25:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:25:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:25:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:25:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:25:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:25:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:25:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name= 2023-12-08T20:25:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 10 2023-12-08T20:25:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:25:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:25:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:25:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=10 2023-12-08T20:25:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:25:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:25:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 45782016 bytes 2023-12-08T20:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 72192000 bytes 2023-12-08T20:25:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 74911744 bytes 2023-12-08T20:25:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 103626752 bytes 2023-12-08T20:25:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 145346560 bytes 2023-12-08T20:25:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 169255936 bytes 2023-12-08T20:25:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 180574208 bytes 2023-12-08T20:25:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 211309568 bytes 2023-12-08T20:25:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 211911680 bytes 2023-12-08T20:25:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 235577344 bytes 2023-12-08T20:25:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 295529472 bytes 2023-12-08T20:25:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 371013632 bytes 2023-12-08T20:25:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 418059264 bytes 2023-12-08T20:25:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 443284480 bytes 2023-12-08T20:25:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 481606656 bytes 2023-12-08T20:25:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 511101952 bytes 2023-12-08T20:25:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 557296640 bytes 2023-12-08T20:25:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 557917184 bytes 2023-12-08T20:25:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 559580160 bytes 2023-12-08T20:25:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 613338112 bytes 2023-12-08T20:25:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 660276224 bytes 2023-12-08T20:25:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 677232640 bytes 2023-12-08T20:25:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 721691648 bytes 2023-12-08T20:25:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 757843968 bytes 2023-12-08T20:25:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 765641728 bytes 2023-12-08T20:25:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 779820032 bytes 2023-12-08T20:25:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 794359808 bytes 2023-12-08T20:25:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 836414464 bytes 2023-12-08T20:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 893624320 bytes 2023-12-08T20:25:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 909875200 bytes 2023-12-08T20:25:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1043231744 bytes 2023-12-08T20:25:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1064630272 bytes 2023-12-08T20:25:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1107551232 bytes 2023-12-08T20:25:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1149246464 bytes 2023-12-08T20:25:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1157476352 bytes 2023-12-08T20:25:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1271478272 bytes 2023-12-08T20:25:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1272105984 bytes 2023-12-08T20:25:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1286717440 bytes 2023-12-08T20:25:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1354905600 bytes 2023-12-08T20:25:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1379777536 bytes 2023-12-08T20:25:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1393584128 bytes 2023-12-08T20:25:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1411137536 bytes 2023-12-08T20:25:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1442695168 bytes 2023-12-08T20:25:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1451305984 bytes 2023-12-08T20:25:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1531094016 bytes 2023-12-08T20:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1531697152 bytes 2023-12-08T20:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1588225024 bytes 2023-12-08T20:25:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1685817344 bytes 2023-12-08T20:25:45.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1720420352 bytes 2023-12-08T20:25:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 1735700480 bytes 2023-12-08T20:25:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 1796925440 bytes 2023-12-08T20:25:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 1838167040 bytes 2023-12-08T20:25:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 1853126656 bytes 2023-12-08T20:25:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 1878318080 bytes 2023-12-08T20:25:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 1884956672 bytes 2023-12-08T20:25:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 1918834688 bytes 2023-12-08T20:25:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 1950744576 bytes 2023-12-08T20:25:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 1996998656 bytes 2023-12-08T20:25:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2023268352 bytes 2023-12-08T20:25:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2081594368 bytes 2023-12-08T20:25:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2172796928 bytes 2023-12-08T20:25:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2254304256 bytes 2023-12-08T20:25:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2297460736 bytes 2023-12-08T20:25:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2298519552 bytes 2023-12-08T20:25:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2334485504 bytes 2023-12-08T20:25:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2411674624 bytes 2023-12-08T20:25:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2453283840 bytes 2023-12-08T20:25:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2462706688 bytes 2023-12-08T20:25:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 2463329280 bytes 2023-12-08T20:25:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 2485871616 bytes 2023-12-08T20:25:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 2539406336 bytes 2023-12-08T20:25:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 2574858240 bytes 2023-12-08T20:25:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 2621829120 bytes 2023-12-08T20:25:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 2622444544 bytes 2023-12-08T20:25:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 2678865920 bytes 2023-12-08T20:25:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 2735298560 bytes 2023-12-08T20:25:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 2903719936 bytes 2023-12-08T20:25:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 2942239744 bytes 2023-12-08T20:25:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 2982771712 bytes 2023-12-08T20:25:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 2993143808 bytes 2023-12-08T20:25:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3021396992 bytes 2023-12-08T20:25:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3049244672 bytes 2023-12-08T20:25:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3088446464 bytes 2023-12-08T20:25:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8400 IOs, 3100121088 bytes 2023-12-08T20:25:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8500 IOs, 3113074688 bytes 2023-12-08T20:25:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8600 IOs, 3201956864 bytes 2023-12-08T20:25:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8700 IOs, 3222504448 bytes 2023-12-08T20:25:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8800 IOs, 3227423744 bytes 2023-12-08T20:25:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8900 IOs, 3283880960 bytes 2023-12-08T20:25:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9000 IOs, 3303892992 bytes 2023-12-08T20:25:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9100 IOs, 3333229568 bytes 2023-12-08T20:25:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9200 IOs, 3349203968 bytes 2023-12-08T20:25:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9300 IOs, 3380737024 bytes 2023-12-08T20:25:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9400 IOs, 3412240384 bytes 2023-12-08T20:25:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9500 IOs, 3413620736 bytes 2023-12-08T20:25:50.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2989160266488382) > weight (0.25) 2023-12-08T20:25:50.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.884503963396301) > weight (0.25) 2023-12-08T20:25:50.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5433924779921897) > weight (0.25) 2023-12-08T20:25:50.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5140410019754698) > weight (0.25) 2023-12-08T20:25:50.318 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-08T20:25:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9600 IOs, 3452371968 bytes 2023-12-08T20:25:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9700 IOs, 3495118848 bytes 2023-12-08T20:25:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:25:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:25:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:25:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:25:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:25:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:25:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:25:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:25:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:25:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:25:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:25:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:25:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:25:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:25:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:25:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:25:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:25:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:25:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:25:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:25:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:25:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:25:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:25:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:25:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 1 2023-12-08T20:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 2 2023-12-08T20:25:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:25:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:25:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:25:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:25:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:25:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:25:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 1 2023-12-08T20:25:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 4 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:25:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:25:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:25:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:25:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:25:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:25:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:25:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 1 2023-12-08T20:25:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 8 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:26:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:26:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:26:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:26:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: down+unknown 2023-12-08T20:26:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: status not found 2023-12-08T20:26:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08T20:26:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 1 2023-12-08T20:26:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 8 2023-12-08T20:26:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:26:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:26:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:26:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702067131,"remote_snapshot_timestamp":1702067131,"replay_state":"idle"} 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:26:07 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:26:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:26:07 2023-12-08T20:26:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:26:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702067131,"remote_snapshot_timestamp":1702067131,"replay_state":"idle"} 2023-12-08T20:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:26:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:26:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:26:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:26:07 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:26:07 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:26:07 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:26:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:26:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:26:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:26:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:26:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:26:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:26:07 2023-12-08T20:26:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:26:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:26:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:26:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:26:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap1 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap1 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap1 2023-12-08T20:26:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:26:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:26:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:26:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:26:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:26:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:26:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:26:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:26:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:26:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:26:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:26:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:26:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:26:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:26:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:26:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:26:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:26:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:26:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:26:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:26:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:26:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:26:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:26:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:26:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:26:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:26:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:26:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:26:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:26:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:26:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:26:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:26:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:26:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:26:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:26:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:26:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:26:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:26:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:26:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:26:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:26:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:26:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:26:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:26:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:26:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 6 2023-12-08T20:26:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:26:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:26:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:26:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:26:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:26:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=6 2023-12-08T20:26:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:26:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:26:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:26:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:26:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:26:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:26:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:26:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:26:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:26:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:26:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 6 = 2023-12-08T20:26:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:26:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:26:23.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:26:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:26:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 6 = 2023-12-08T20:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:26:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:26:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:26:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:26:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:26:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:26:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:26:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 6 = 2023-12-08T20:26:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:26:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:26:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:26:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=6 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 6 = 6 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap1 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:26:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:26:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:26:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:26:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:26:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap1 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap1 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:26:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:26:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:26:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:26:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:26:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:26:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap1-clean 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap1-clean 2023-12-08T20:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:26:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:26:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:26:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=12 2023-12-08T20:26:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:26:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:26:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 27832320 bytes 2023-12-08T20:26:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 73515008 bytes 2023-12-08T20:26:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 95499264 bytes 2023-12-08T20:26:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 104607744 bytes 2023-12-08T20:26:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 120710144 bytes 2023-12-08T20:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 150847488 bytes 2023-12-08T20:26:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 180701184 bytes 2023-12-08T20:26:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 278499328 bytes 2023-12-08T20:26:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 333514752 bytes 2023-12-08T20:26:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 391438336 bytes 2023-12-08T20:26:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 436914176 bytes 2023-12-08T20:26:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 466341888 bytes 2023-12-08T20:26:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 503147520 bytes 2023-12-08T20:26:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 538186752 bytes 2023-12-08T20:26:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 605454336 bytes 2023-12-08T20:26:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 623485952 bytes 2023-12-08T20:26:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 701145088 bytes 2023-12-08T20:26:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 717413376 bytes 2023-12-08T20:26:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 750754816 bytes 2023-12-08T20:26:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 780930048 bytes 2023-12-08T20:26:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 785936384 bytes 2023-12-08T20:26:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 840023040 bytes 2023-12-08T20:26:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 878236672 bytes 2023-12-08T20:26:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 923249664 bytes 2023-12-08T20:26:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 928745472 bytes 2023-12-08T20:26:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 986112000 bytes 2023-12-08T20:26:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1093223424 bytes 2023-12-08T20:26:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1132349440 bytes 2023-12-08T20:26:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1171526656 bytes 2023-12-08T20:26:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1210142720 bytes 2023-12-08T20:26:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1241076736 bytes 2023-12-08T20:26:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1325383680 bytes 2023-12-08T20:26:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1404666880 bytes 2023-12-08T20:26:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1477581824 bytes 2023-12-08T20:26:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1553314816 bytes 2023-12-08T20:26:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1555960832 bytes 2023-12-08T20:26:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1592131584 bytes 2023-12-08T20:26:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1645748224 bytes 2023-12-08T20:26:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1713301504 bytes 2023-12-08T20:26:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1742856192 bytes 2023-12-08T20:26:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1800483840 bytes 2023-12-08T20:26:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1881575424 bytes 2023-12-08T20:26:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1931435008 bytes 2023-12-08T20:26:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1958792192 bytes 2023-12-08T20:26:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1963345920 bytes 2023-12-08T20:26:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2007228416 bytes 2023-12-08T20:26:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2084534272 bytes 2023-12-08T20:26:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2167256064 bytes 2023-12-08T20:26:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2240166912 bytes 2023-12-08T20:26:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2321040384 bytes 2023-12-08T20:26:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2355244032 bytes 2023-12-08T20:26:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2371840000 bytes 2023-12-08T20:26:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2390594560 bytes 2023-12-08T20:26:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2435652608 bytes 2023-12-08T20:26:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2485927936 bytes 2023-12-08T20:26:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2486540288 bytes 2023-12-08T20:26:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2509750272 bytes 2023-12-08T20:26:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2547467264 bytes 2023-12-08T20:26:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2572879872 bytes 2023-12-08T20:26:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2613512192 bytes 2023-12-08T20:26:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2662642688 bytes 2023-12-08T20:26:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2724569088 bytes 2023-12-08T20:26:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2756988928 bytes 2023-12-08T20:26:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2779304960 bytes 2023-12-08T20:26:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2835964928 bytes 2023-12-08T20:26:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2857664512 bytes 2023-12-08T20:26:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2883414016 bytes 2023-12-08T20:26:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2906125312 bytes 2023-12-08T20:26:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 2938828800 bytes 2023-12-08T20:26:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 2956427264 bytes 2023-12-08T20:26:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3016141824 bytes 2023-12-08T20:26:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3062511616 bytes 2023-12-08T20:26:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3108660224 bytes 2023-12-08T20:26:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3176261632 bytes 2023-12-08T20:26:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3276934144 bytes 2023-12-08T20:26:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3346941952 bytes 2023-12-08T20:26:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3376681984 bytes 2023-12-08T20:26:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3418491904 bytes 2023-12-08T20:26:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3483929600 bytes 2023-12-08T20:26:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3528676352 bytes 2023-12-08T20:26:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3552189440 bytes 2023-12-08T20:26:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3597208576 bytes 2023-12-08T20:26:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3624548352 bytes 2023-12-08T20:26:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8400 IOs, 3636942848 bytes 2023-12-08T20:26:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8500 IOs, 3711797248 bytes 2023-12-08T20:26:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8600 IOs, 3739607040 bytes 2023-12-08T20:26:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8700 IOs, 3758373888 bytes 2023-12-08T20:26:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8800 IOs, 3836181504 bytes 2023-12-08T20:26:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8900 IOs, 3846733824 bytes 2023-12-08T20:26:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9000 IOs, 3897265152 bytes 2023-12-08T20:26:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9100 IOs, 3999414272 bytes 2023-12-08T20:26:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9200 IOs, 4053918720 bytes 2023-12-08T20:26:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9300 IOs, 4149899264 bytes 2023-12-08T20:26:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9400 IOs, 4200604672 bytes 2023-12-08T20:26:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9500 IOs, 4254651392 bytes 2023-12-08T20:26:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9600 IOs, 4345824256 bytes 2023-12-08T20:26:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9700 IOs, 4394362880 bytes 2023-12-08T20:26:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9800 IOs, 4440185856 bytes 2023-12-08T20:26:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9900 IOs, 4475687936 bytes 2023-12-08T20:26:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:10000 IOs, 4535184384 bytes 2023-12-08T20:26:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:10100 IOs, 4584102912 bytes 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:26:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:26:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:26:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:26:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":17856819.2,"bytes_per_snapshot":267852288.0,"last_snapshot_bytes":535704576,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067182,"remote_snapshot_timestamp":1702067182,"replay_state":"idle"} 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:26:37 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:26:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:26:37 2023-12-08T20:26:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:26:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":17856819.2,"bytes_per_snapshot":267852288.0,"last_snapshot_bytes":535704576,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067182,"remote_snapshot_timestamp":1702067182,"replay_state":"idle"} 2023-12-08T20:26:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:26:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:26:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:26:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:26:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:26:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:26:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:26:37 2023-12-08T20:26:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:26:37 2023-12-08T20:26:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:26:37 2023-12-08T20:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:26:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:26:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:26:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:26:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:26:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:26:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:26:37 2023-12-08T20:26:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:26:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:26:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:26:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:26:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:26:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:26:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap2 2023-12-08T20:26:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap2 2023-12-08T20:26:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:26:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:26:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:26:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap2 2023-12-08T20:26:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:26:46.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6113499014144919) > weight (0.25) 2023-12-08T20:26:46.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6212924815753161) > weight (0.25) 2023-12-08T20:26:46.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7155801873289778) > weight (0.25) 2023-12-08T20:26:46.329 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-08T20:26:46.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-08T20:26:46.330 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-08T20:26:46.330 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-08T20:26:46.330 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:26:46.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:26:46.325+0000 7fbb238ae700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 34510) UID: 1000 2023-12-08T20:26:46.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:26:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:26:52.432 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-08T20:26:52.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-08T20:26:52.432 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-08T20:26:52.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-08T20:26:52.436 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-08T20:27:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:27:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:27:13.320 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8817523277797823) > weight (0.25) 2023-12-08T20:27:13.320 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5354352488164082) > weight (0.25) 2023-12-08T20:27:13.320 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-08T20:27:13.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-08T20:27:13.320 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-08T20:27:13.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi158.stderr:2023-12-08T20:27:13.316+0000 7f24712a0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 34345) UID: 1000 2023-12-08T20:27:13.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:27:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:27:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ greplocal -i cluster=cluster1 state 2023-12-08T20:27:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:27:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:27:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:27:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:27:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:27:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:27:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:27:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:27:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:27:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:27:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:27:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:27:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:27:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:27:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:27:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:27:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:27:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:27:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:27:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:27:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:27:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:27:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:27:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:27:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:27:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:27:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:27:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:27:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:27:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:27:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:27:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:27:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:27:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:27:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:27:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:27:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:27:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:27:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:27:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:27:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:27:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:27:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:27:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:27:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:27:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 9 2023-12-08T20:27:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:27:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:27:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:27:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:27:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:27:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:27:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=9 2023-12-08T20:27:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:27:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:27:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:27:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:27:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:27:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:27:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:27:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:27:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:27:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:27:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 9 = 2023-12-08T20:27:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:27:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:27:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:27:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:27:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:27:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:27:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:27:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:27:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:27:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 9 = 2023-12-08T20:27:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:27:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:27:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:27:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:27:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:27:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:27:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:27:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:27:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:27:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 9 = 2023-12-08T20:27:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:27:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:27:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:27:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:27:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:27:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=9 2023-12-08T20:27:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 9 = 9 2023-12-08T20:27:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-08T20:27:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap2 2023-12-08T20:27:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:27:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:27:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-08T20:27:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:27:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:27:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:27:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 4 2023-12-08T20:27:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:27:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:27:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:27:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:27:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:27:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:27:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:27:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:27:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:27:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:27:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:27:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:27:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:27:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:27:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap1-clean ] 2023-12-08T20:27:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-08T20:27:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap1-clean 2023-12-08T20:27:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:27:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:27:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:27:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:27:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:27:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:22.321 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-08T20:27:22.321 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:27:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:27:22.322 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-08T20:27:22.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-08T20:27:22.322 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-08T20:27:22.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-08T20:27:22.325 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-08T20:27:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap2 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap2 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:27:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:27:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:27:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap2-clean 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap2-clean 2023-12-08T20:27:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:27:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:27:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:27:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:27:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:27:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:27:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:27:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 12917760 bytes 2023-12-08T20:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 62439424 bytes 2023-12-08T20:27:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 131969024 bytes 2023-12-08T20:27:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 160452608 bytes 2023-12-08T20:27:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 170458112 bytes 2023-12-08T20:27:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 192750592 bytes 2023-12-08T20:27:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 253562880 bytes 2023-12-08T20:27:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 312327168 bytes 2023-12-08T20:27:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 391337984 bytes 2023-12-08T20:27:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 444199936 bytes 2023-12-08T20:27:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 547510272 bytes 2023-12-08T20:27:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 548124672 bytes 2023-12-08T20:27:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 640852992 bytes 2023-12-08T20:27:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 660464640 bytes 2023-12-08T20:27:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 694612992 bytes 2023-12-08T20:27:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 714482688 bytes 2023-12-08T20:27:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 752034816 bytes 2023-12-08T20:27:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 784704512 bytes 2023-12-08T20:27:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 805976064 bytes 2023-12-08T20:27:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 822176768 bytes 2023-12-08T20:27:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 857918464 bytes 2023-12-08T20:27:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 890718208 bytes 2023-12-08T20:27:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 934022144 bytes 2023-12-08T20:27:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 940078080 bytes 2023-12-08T20:27:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 982137856 bytes 2023-12-08T20:27:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 993792000 bytes 2023-12-08T20:27:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 996054016 bytes 2023-12-08T20:27:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1049559040 bytes 2023-12-08T20:27:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1059336192 bytes 2023-12-08T20:27:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1128029184 bytes 2023-12-08T20:27:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1160016896 bytes 2023-12-08T20:27:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1244703744 bytes 2023-12-08T20:27:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1302349824 bytes 2023-12-08T20:27:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1343373312 bytes 2023-12-08T20:27:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1466001408 bytes 2023-12-08T20:27:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1513066496 bytes 2023-12-08T20:27:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1602411520 bytes 2023-12-08T20:27:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1685409792 bytes 2023-12-08T20:27:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1762944000 bytes 2023-12-08T20:27:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1785835520 bytes 2023-12-08T20:27:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1835546624 bytes 2023-12-08T20:27:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1881432064 bytes 2023-12-08T20:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1930090496 bytes 2023-12-08T20:27:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2006302720 bytes 2023-12-08T20:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2084917248 bytes 2023-12-08T20:27:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2131241984 bytes 2023-12-08T20:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2131837952 bytes 2023-12-08T20:27:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2165297152 bytes 2023-12-08T20:27:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2193241088 bytes 2023-12-08T20:27:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2249035776 bytes 2023-12-08T20:27:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2311387136 bytes 2023-12-08T20:27:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2341974016 bytes 2023-12-08T20:27:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2377967616 bytes 2023-12-08T20:27:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2441162752 bytes 2023-12-08T20:27:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2474051584 bytes 2023-12-08T20:27:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2511100928 bytes 2023-12-08T20:27:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2599655424 bytes 2023-12-08T20:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2626407424 bytes 2023-12-08T20:27:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2655566848 bytes 2023-12-08T20:27:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2669172736 bytes 2023-12-08T20:27:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2729625600 bytes 2023-12-08T20:27:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2755161088 bytes 2023-12-08T20:27:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2779971584 bytes 2023-12-08T20:27:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2840585216 bytes 2023-12-08T20:27:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2907495424 bytes 2023-12-08T20:27:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2950546432 bytes 2023-12-08T20:27:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2954324992 bytes 2023-12-08T20:27:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2978407424 bytes 2023-12-08T20:27:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3075014656 bytes 2023-12-08T20:27:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3122323456 bytes 2023-12-08T20:27:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3173041152 bytes 2023-12-08T20:27:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3218120704 bytes 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:27:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:27:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:27:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:27:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":267852288.0,"last_snapshot_bytes":535704576,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067182,"remote_snapshot_timestamp":1702067182,"replay_state":"idle"} 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:27:07 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:27:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:27:07 2023-12-08T20:27:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:27:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":267852288.0,"last_snapshot_bytes":535704576,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067182,"remote_snapshot_timestamp":1702067182,"replay_state":"idle"} 2023-12-08T20:27:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:27:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:27:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:27:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:27:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:27:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:27:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:27:07 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:27:07 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:27:07 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:27:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:27:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:27:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:27:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:27:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:27:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:27:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:27:07 2023-12-08T20:27:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:27:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:27:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + echo up+replaying 2023-12-08T20:27:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep up+replaying 2023-12-08T20:27:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:27:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:27:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap3 2023-12-08T20:27:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap3 2023-12-08T20:27:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:27:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:27:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:27:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap3 2023-12-08T20:27:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:28:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:28:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:28:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:28:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:28:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:28:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:28:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:28:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:28:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:28:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:28:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:28:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:28:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state+ 2023-12-08T20:28:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:local cluster=cluster1 2023-12-08T20:28:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:28:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:28:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:28:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:28:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:28:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:28:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:28:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:28:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:28:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:28:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:28:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:28:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:28:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:28:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:28:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:28:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:28:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:28:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:28:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:28:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:28:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:28:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:28:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:28:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:28:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:28:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:28:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:28:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:28:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:28:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:28:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:28:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:28:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:28:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:28:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:28:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:28:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:28:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:28:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:28:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:28:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:28:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:28:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:28:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:28:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:28:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:28:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:28:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:28:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:28:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:28:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:28:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:28:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:28:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:28:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:28:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:28:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:28:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:28:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:28:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 13 2023-12-08T20:28:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:28:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:28:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:28:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:28:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=13 2023-12-08T20:28:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:28:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:28:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:28:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:28:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:28:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:28:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 13 = 2023-12-08T20:28:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:28:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:28:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:28:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:28:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:28:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:28:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:28:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 13 = 2023-12-08T20:28:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:28:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:28:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:28:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:28:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:28:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:28:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:28:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 13 = 2023-12-08T20:28:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:28:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:28:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=13 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 13 = 13 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap3 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:28:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:28:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 6 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:28:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:27:21 2023 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:28:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:28:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap2-clean ] 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap2-clean 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:28:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:28:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:28:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap3 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap3 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:28:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:28:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:28:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap3-clean 2023-12-08T20:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-08T20:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap3-clean 2023-12-08T20:28:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:28:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:28:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:28:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=8 2023-12-08T20:28:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:28:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:28:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 105543680 bytes 2023-12-08T20:28:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 144335872 bytes 2023-12-08T20:28:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 180758528 bytes 2023-12-08T20:28:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 199133184 bytes 2023-12-08T20:28:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 279750656 bytes 2023-12-08T20:28:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 280373248 bytes 2023-12-08T20:28:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 289740800 bytes 2023-12-08T20:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 334620672 bytes 2023-12-08T20:28:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 382272512 bytes 2023-12-08T20:28:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 416763904 bytes 2023-12-08T20:28:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 465474560 bytes 2023-12-08T20:28:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 560747520 bytes 2023-12-08T20:28:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 607474688 bytes 2023-12-08T20:28:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 631360512 bytes 2023-12-08T20:28:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 660785152 bytes 2023-12-08T20:28:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 769819648 bytes 2023-12-08T20:28:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 782357504 bytes 2023-12-08T20:28:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 803608576 bytes 2023-12-08T20:28:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 807443456 bytes 2023-12-08T20:28:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 858442752 bytes 2023-12-08T20:28:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 873409536 bytes 2023-12-08T20:28:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 928518144 bytes 2023-12-08T20:28:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1019868160 bytes 2023-12-08T20:28:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1062480896 bytes 2023-12-08T20:28:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1087536128 bytes 2023-12-08T20:28:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1116171264 bytes 2023-12-08T20:28:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1136186368 bytes 2023-12-08T20:28:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1151298560 bytes 2023-12-08T20:28:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1174768640 bytes 2023-12-08T20:28:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1181910016 bytes 2023-12-08T20:28:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1264180224 bytes 2023-12-08T20:28:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1284277248 bytes 2023-12-08T20:28:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1332658176 bytes 2023-12-08T20:28:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1349676032 bytes 2023-12-08T20:28:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1377991680 bytes 2023-12-08T20:28:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1468061696 bytes 2023-12-08T20:28:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1511598080 bytes 2023-12-08T20:28:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1539368960 bytes 2023-12-08T20:28:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1594157056 bytes 2023-12-08T20:28:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1693787136 bytes 2023-12-08T20:28:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1726099456 bytes 2023-12-08T20:28:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1749155840 bytes 2023-12-08T20:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1788217344 bytes 2023-12-08T20:28:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1842813952 bytes 2023-12-08T20:28:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1897007104 bytes 2023-12-08T20:28:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1924699136 bytes 2023-12-08T20:28:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1963265024 bytes 2023-12-08T20:28:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2046653440 bytes 2023-12-08T20:28:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2134477824 bytes 2023-12-08T20:28:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2175504384 bytes 2023-12-08T20:28:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2178080768 bytes 2023-12-08T20:28:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2287863808 bytes 2023-12-08T20:28:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2332000256 bytes 2023-12-08T20:28:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2423038976 bytes 2023-12-08T20:28:23.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9337386768840461) > weight (0.25) 2023-12-08T20:28:23.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7929349756656945) > weight (0.25) 2023-12-08T20:28:23.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.33900723298275415) > weight (0.25) 2023-12-08T20:28:23.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6033538642642983) > weight (0.25) 2023-12-08T20:28:23.326 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-08T20:28:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2491030528 bytes 2023-12-08T20:28:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2536064000 bytes 2023-12-08T20:28:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2621916160 bytes 2023-12-08T20:28:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2697456640 bytes 2023-12-08T20:28:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2737969152 bytes 2023-12-08T20:28:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2770644992 bytes 2023-12-08T20:28:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2800724992 bytes 2023-12-08T20:28:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2868456448 bytes 2023-12-08T20:28:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2889223168 bytes 2023-12-08T20:28:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:28:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:28:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:28:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:28:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:28:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:28:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:28:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:28:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536161792.0,"last_snapshot_bytes":536619008,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067235,"remote_snapshot_timestamp":1702067235,"replay_state":"idle"} 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:28:07 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:28:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:28:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:28:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:28:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:28:07 2023-12-08T20:28:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:28:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536161792.0,"last_snapshot_bytes":536619008,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067235,"remote_snapshot_timestamp":1702067235,"replay_state":"idle"} 2023-12-08T20:28:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:28:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:28:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:28:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:28:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:28:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:28:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:28:07 2023-12-08T20:28:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:28:07 2023-12-08T20:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:28:07 2023-12-08T20:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:28:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:28:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:28:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:28:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:28:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:28:07 2023-12-08T20:28:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:28:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:28:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:28:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:28:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap4 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap4 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap4 2023-12-08T20:28:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:28:44.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6916746865069388) > weight (0.25) 2023-12-08T20:28:44.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5586569485421604) > weight (0.25) 2023-12-08T20:28:44.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5866438609103406) > weight (0.25) 2023-12-08T20:28:44.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.46102416130735313) > weight (0.25) 2023-12-08T20:28:44.438 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-08T20:28:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:28:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:28:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster=cluster1 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ localgrep instance=cluster1 2023-12-08T20:28:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: -i state+ 2023-12-08T20:28:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:local loop_instance 2023-12-08T20:28:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:28:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:28:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:28:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:28:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:28:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:28:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:28:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:28:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:28:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:28:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:28:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:28:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:28:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:28:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:28:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:28:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:28:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:28:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:28:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:28:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:28:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:28:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:28:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:28:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:28:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:28:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:28:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:28:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:28:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:28:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:28:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:28:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:28:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:28:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:28:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:28:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:28:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:28:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:28:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:28:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:28:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:28:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:28:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:28:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 17 2023-12-08T20:28:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:28:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:28:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:28:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:28:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:28:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=17 2023-12-08T20:28:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:28:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:28:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:28:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:28:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 17 = 2023-12-08T20:28:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:28:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:28:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:28:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:28:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:28:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:28:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:28:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 17 = 2023-12-08T20:28:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:28:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:28:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:28:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:28:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:28:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:28:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:28:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:28:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:28:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:28:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:28:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:28:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:28:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 17 = 2023-12-08T20:28:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:29:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:29:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:29:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=17 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 17 = 17 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap4 2023-12-08T20:29:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:29:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:29:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-08T20:29:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:29:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:29:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:29:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 8 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:27:21 2023 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:29:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:29:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap3-clean ] 2023-12-08T20:29:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap3-clean 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap4 2023-12-08T20:29:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap4 2023-12-08T20:29:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:29:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap4-clean 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap4-clean 2023-12-08T20:29:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:29:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:29:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:29:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:29:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:29:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:29:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 39401472 bytes 2023-12-08T20:29:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 100984832 bytes 2023-12-08T20:29:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 133792768 bytes 2023-12-08T20:29:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 190679040 bytes 2023-12-08T20:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 234563584 bytes 2023-12-08T20:29:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 290179072 bytes 2023-12-08T20:29:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 326199296 bytes 2023-12-08T20:29:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 406219776 bytes 2023-12-08T20:29:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 437697536 bytes 2023-12-08T20:29:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 481678336 bytes 2023-12-08T20:29:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 570412032 bytes 2023-12-08T20:29:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 612566016 bytes 2023-12-08T20:29:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 709694464 bytes 2023-12-08T20:29:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 800373760 bytes 2023-12-08T20:29:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 817614848 bytes 2023-12-08T20:29:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 853310464 bytes 2023-12-08T20:29:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 912028672 bytes 2023-12-08T20:29:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 957080576 bytes 2023-12-08T20:29:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 1002982400 bytes 2023-12-08T20:29:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1021674496 bytes 2023-12-08T20:29:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1035111424 bytes 2023-12-08T20:29:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1099544576 bytes 2023-12-08T20:29:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1154248704 bytes 2023-12-08T20:29:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1220946944 bytes 2023-12-08T20:29:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1275036672 bytes 2023-12-08T20:29:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1319812096 bytes 2023-12-08T20:29:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1398548480 bytes 2023-12-08T20:29:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1443868672 bytes 2023-12-08T20:29:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1475076096 bytes 2023-12-08T20:29:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1522182144 bytes 2023-12-08T20:29:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1570326528 bytes 2023-12-08T20:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1625149440 bytes 2023-12-08T20:29:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1662804992 bytes 2023-12-08T20:29:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1743609856 bytes 2023-12-08T20:29:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1776307200 bytes 2023-12-08T20:29:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1805977600 bytes 2023-12-08T20:29:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1861252096 bytes 2023-12-08T20:29:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1917805568 bytes 2023-12-08T20:29:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1926528000 bytes 2023-12-08T20:29:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1973308416 bytes 2023-12-08T20:29:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1985868800 bytes 2023-12-08T20:29:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 2033848320 bytes 2023-12-08T20:29:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2059478016 bytes 2023-12-08T20:29:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2090023936 bytes 2023-12-08T20:29:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2090647552 bytes 2023-12-08T20:29:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2097142784 bytes 2023-12-08T20:29:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2149830656 bytes 2023-12-08T20:29:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2215791616 bytes 2023-12-08T20:29:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2251938816 bytes 2023-12-08T20:29:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2336749568 bytes 2023-12-08T20:29:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2338898944 bytes 2023-12-08T20:29:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2383831040 bytes 2023-12-08T20:29:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2407563264 bytes 2023-12-08T20:29:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2447538176 bytes 2023-12-08T20:29:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2503942144 bytes 2023-12-08T20:29:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2545825792 bytes 2023-12-08T20:29:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2576073728 bytes 2023-12-08T20:29:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2644313088 bytes 2023-12-08T20:29:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2701841408 bytes 2023-12-08T20:29:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2741851136 bytes 2023-12-08T20:29:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2787564544 bytes 2023-12-08T20:29:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2788176896 bytes 2023-12-08T20:29:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2861542400 bytes 2023-12-08T20:29:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2913272832 bytes 2023-12-08T20:29:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2927739904 bytes 2023-12-08T20:29:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2994869248 bytes 2023-12-08T20:29:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 3063934976 bytes 2023-12-08T20:29:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:29:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:29:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:29:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":17820125.87,"bytes_per_snapshot":533478400.0,"last_snapshot_bytes":534603776,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067335,"remote_snapshot_timestamp":1702067335,"replay_state":"idle"} 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:29:07 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:29:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:29:07 2023-12-08T20:29:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:29:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":17820125.87,"bytes_per_snapshot":533478400.0,"last_snapshot_bytes":534603776,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067335,"remote_snapshot_timestamp":1702067335,"replay_state":"idle"} 2023-12-08T20:29:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:29:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + echo up+replaying 2023-12-08T20:29:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep ^up+ 2023-12-08T20:29:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:29:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:29:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:29:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:29:07 2023-12-08T20:29:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:29:07 2023-12-08T20:29:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:29:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:29:07 2023-12-08T20:29:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:29:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:29:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:29:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:29:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:29:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:29:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:29:07 2023-12-08T20:29:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:29:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:29:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:29:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:29:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap5 2023-12-08T20:29:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap5 2023-12-08T20:29:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap5 2023-12-08T20:29:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:29:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:29:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:29:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:29:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:29:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:29:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster=cluster1grep -i state 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:29:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:29:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:29:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:29:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:29:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:29:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:29:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:29:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:29:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:29:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:29:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:29:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:29:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:29:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:29:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:29:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:29:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:29:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:29:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:29:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:29:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:29:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:29:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:29:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:29:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:29:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:29:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:29:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:29:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:29:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:29:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:29:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:29:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:29:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:29:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:29:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:29:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:29:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:29:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:29:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:29:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 21 2023-12-08T20:29:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:29:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:29:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:29:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:29:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=21 2023-12-08T20:29:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:29:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:29:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:29:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:29:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:29:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:29:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:29:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:29:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:29:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:29:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 21 = 2023-12-08T20:29:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:29:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:29:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:29:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:29:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:29:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:29:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:29:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:29:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:29:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 21 = 2023-12-08T20:29:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:29:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:29:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:29:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:29:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:29:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:29:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:29:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:29:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:29:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 21 = 2023-12-08T20:29:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:29:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:29:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:29:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:29:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:29:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=21 2023-12-08T20:29:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 21 = 21 2023-12-08T20:29:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-08T20:29:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:29:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap5 2023-12-08T20:29:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:29:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:29:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-08T20:29:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 10 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:29:02 2023 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:29:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:29:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:29:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:29:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap4-clean ] 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap4-clean 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap5 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap5 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:29:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:55.439 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2919394274842918) > weight (0.25) 2023-12-08T20:29:55.439 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9580594618813117) > weight (0.25) 2023-12-08T20:29:55.439 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8849842239261138) > weight (0.25) 2023-12-08T20:29:55.439 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-08T20:29:55.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-08T20:29:55.440 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-08T20:29:55.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:29:55.433+0000 7fc5d8c59700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 37053) UID: 1000 2023-12-08T20:29:55.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:29:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:29:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:29:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:29:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap5-clean 2023-12-08T20:29:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-08T20:29:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap5-clean 2023-12-08T20:29:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:29:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:29:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=13 2023-12-08T20:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:29:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:29:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 50110464 bytes 2023-12-08T20:29:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 74997760 bytes 2023-12-08T20:29:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 101895168 bytes 2023-12-08T20:29:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 102501376 bytes 2023-12-08T20:29:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 120418304 bytes 2023-12-08T20:29:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 195495936 bytes 2023-12-08T20:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 256187392 bytes 2023-12-08T20:29:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 317919232 bytes 2023-12-08T20:29:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 334264320 bytes 2023-12-08T20:29:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 341743616 bytes 2023-12-08T20:29:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 422028288 bytes 2023-12-08T20:29:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 422629376 bytes 2023-12-08T20:29:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 423214080 bytes 2023-12-08T20:29:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 521294848 bytes 2023-12-08T20:29:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 596661248 bytes 2023-12-08T20:29:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 662906880 bytes 2023-12-08T20:29:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 686322688 bytes 2023-12-08T20:29:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 719065088 bytes 2023-12-08T20:29:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 769949696 bytes 2023-12-08T20:29:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 803204096 bytes 2023-12-08T20:29:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 903177216 bytes 2023-12-08T20:29:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 930972672 bytes 2023-12-08T20:29:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 974777344 bytes 2023-12-08T20:29:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1023941632 bytes 2023-12-08T20:30:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1100422144 bytes 2023-12-08T20:30:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1116277760 bytes 2023-12-08T20:30:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1175436288 bytes 2023-12-08T20:30:00.441 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-08T20:30:00.441 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:30:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:30:00.442 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-08T20:30:00.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-08T20:30:00.442 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-08T20:30:00.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-08T20:30:00.445 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2023-12-08T20:30:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1219827712 bytes 2023-12-08T20:30:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1271334912 bytes 2023-12-08T20:30:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1314851840 bytes 2023-12-08T20:30:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1347538944 bytes 2023-12-08T20:30:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1416556544 bytes 2023-12-08T20:30:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1468305408 bytes 2023-12-08T20:30:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1529803776 bytes 2023-12-08T20:30:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1562368000 bytes 2023-12-08T20:30:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1646523392 bytes 2023-12-08T20:30:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1648622592 bytes 2023-12-08T20:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1740660736 bytes 2023-12-08T20:30:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1751537664 bytes 2023-12-08T20:30:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1834284032 bytes 2023-12-08T20:30:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1858181120 bytes 2023-12-08T20:30:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1912841216 bytes 2023-12-08T20:30:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1976341504 bytes 2023-12-08T20:30:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1986180096 bytes 2023-12-08T20:30:02.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2077503488 bytes 2023-12-08T20:30:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2156886016 bytes 2023-12-08T20:30:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2208575488 bytes 2023-12-08T20:30:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2214075392 bytes 2023-12-08T20:30:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2241169408 bytes 2023-12-08T20:30:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2245408768 bytes 2023-12-08T20:30:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2277488640 bytes 2023-12-08T20:30:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2355795968 bytes 2023-12-08T20:30:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2383244288 bytes 2023-12-08T20:30:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2428098560 bytes 2023-12-08T20:30:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2439498752 bytes 2023-12-08T20:30:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2510430208 bytes 2023-12-08T20:30:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2569939968 bytes 2023-12-08T20:30:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2581239808 bytes 2023-12-08T20:30:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2639453184 bytes 2023-12-08T20:30:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2679685120 bytes 2023-12-08T20:30:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2717690880 bytes 2023-12-08T20:30:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2771182592 bytes 2023-12-08T20:30:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2870233088 bytes 2023-12-08T20:30:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2915141632 bytes 2023-12-08T20:30:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2973771776 bytes 2023-12-08T20:30:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 3054722048 bytes 2023-12-08T20:30:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 3081789440 bytes 2023-12-08T20:30:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3117209600 bytes 2023-12-08T20:30:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3127014400 bytes 2023-12-08T20:30:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3188196352 bytes 2023-12-08T20:30:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3227134976 bytes 2023-12-08T20:30:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3279289344 bytes 2023-12-08T20:30:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3290103808 bytes 2023-12-08T20:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3304707072 bytes 2023-12-08T20:30:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3349582848 bytes 2023-12-08T20:30:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3398423552 bytes 2023-12-08T20:30:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3455696896 bytes 2023-12-08T20:30:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3456320512 bytes 2023-12-08T20:30:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3456932864 bytes 2023-12-08T20:30:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3528962048 bytes 2023-12-08T20:30:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3591753728 bytes 2023-12-08T20:30:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3663982592 bytes 2023-12-08T20:30:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3675713536 bytes 2023-12-08T20:30:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8400 IOs, 3690682368 bytes 2023-12-08T20:30:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8500 IOs, 3751232512 bytes 2023-12-08T20:30:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8600 IOs, 3765720064 bytes 2023-12-08T20:30:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8700 IOs, 3850536960 bytes 2023-12-08T20:30:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8800 IOs, 3861647360 bytes 2023-12-08T20:30:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8900 IOs, 3918608384 bytes 2023-12-08T20:30:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9000 IOs, 3996931072 bytes 2023-12-08T20:30:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9100 IOs, 4064212992 bytes 2023-12-08T20:30:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9200 IOs, 4104590336 bytes 2023-12-08T20:30:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9300 IOs, 4148044800 bytes 2023-12-08T20:30:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9400 IOs, 4208025600 bytes 2023-12-08T20:30:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9500 IOs, 4266131456 bytes 2023-12-08T20:30:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9600 IOs, 4283751424 bytes 2023-12-08T20:30:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9700 IOs, 4334011392 bytes 2023-12-08T20:30:09.327 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9589914252906853) > weight (0.25) 2023-12-08T20:30:09.327 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9240090140928312) > weight (0.25) 2023-12-08T20:30:09.327 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.790868954944239) > weight (0.25) 2023-12-08T20:30:09.327 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9656753275262747) > weight (0.25) 2023-12-08T20:30:09.327 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-08T20:30:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9800 IOs, 4365326336 bytes 2023-12-08T20:30:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9900 IOs, 4421234688 bytes 2023-12-08T20:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:30:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:30:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:30:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:30:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:30:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:30:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":17744145.07,"bytes_per_snapshot":533464064.0,"last_snapshot_bytes":532324352,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067386,"remote_snapshot_timestamp":1702067386,"replay_state":"idle"} 2023-12-08T20:30:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:30:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:30:07 2023-12-08T20:30:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:30:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:30:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:30:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:30:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:30:07 2023-12-08T20:30:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:30:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":17744145.07,"bytes_per_snapshot":533464064.0,"last_snapshot_bytes":532324352,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067386,"remote_snapshot_timestamp":1702067386,"replay_state":"idle"} 2023-12-08T20:30:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:30:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:30:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:30:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:30:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:30:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:30:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:30:07 2023-12-08T20:30:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:30:07 2023-12-08T20:30:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:30:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:30:07 2023-12-08T20:30:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:30:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:30:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:30:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:30:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:30:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:30:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:30:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:30:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:30:07 2023-12-08T20:30:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:30:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:30:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:30:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:30:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:30:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap6 2023-12-08T20:30:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap6 2023-12-08T20:30:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:30:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap6 2023-12-08T20:30:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:30:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:30:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:30:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:30:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:30:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:30:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:30:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep+ -ilocal state cluster=cluster1 2023-12-08T20:30:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:30:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:30:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:30:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:30:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:30:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:30:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:30:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:30:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:30:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:30:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:30:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:30:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:30:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:30:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:30:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:30:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:30:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:30:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:30:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:30:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:30:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:30:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:30:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:30:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:30:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:30:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:30:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:30:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:30:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:30:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:30:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:30:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:30:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:30:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:30:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:30:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 25 2023-12-08T20:30:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:30:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:30:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:30:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:30:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:30:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:30:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=25 2023-12-08T20:30:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:30:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:30:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:30:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:30:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:30:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:30:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:30:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:30:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 25 = 2023-12-08T20:30:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:30:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:30:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:30:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:30:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:30:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:30:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:30:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:30:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:30:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 25 = 2023-12-08T20:30:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:30:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:30:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:30:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:30:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:30:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:30:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:30:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:30:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:30:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 25 = 2023-12-08T20:30:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:30:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:30:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:30:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:30:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:30:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=25 2023-12-08T20:30:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 25 = 25 2023-12-08T20:30:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-08T20:30:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap6 2023-12-08T20:30:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:30:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:30:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-08T20:30:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:30:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:30:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 12 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:29:02 2023 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap5-clean ] 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap5-clean 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:30:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:30:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:30:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap6 2023-12-08T20:30:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap6 2023-12-08T20:30:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:30:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:30:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:30:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:30:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:30:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap6-clean 2023-12-08T20:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-08T20:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap6-clean 2023-12-08T20:30:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:30:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=7 2023-12-08T20:30:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:30:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:30:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 605184 bytes 2023-12-08T20:30:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 41093120 bytes 2023-12-08T20:30:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 58121216 bytes 2023-12-08T20:30:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 125325312 bytes 2023-12-08T20:30:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 193844224 bytes 2023-12-08T20:30:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 246511616 bytes 2023-12-08T20:30:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 361612288 bytes 2023-12-08T20:30:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 374719488 bytes 2023-12-08T20:30:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 411598848 bytes 2023-12-08T20:30:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 421778432 bytes 2023-12-08T20:30:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 476596224 bytes 2023-12-08T20:30:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 507383808 bytes 2023-12-08T20:30:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 507986944 bytes 2023-12-08T20:30:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 526525440 bytes 2023-12-08T20:30:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 536199168 bytes 2023-12-08T20:30:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 566486016 bytes 2023-12-08T20:30:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 633049088 bytes 2023-12-08T20:30:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 708046848 bytes 2023-12-08T20:30:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 736264192 bytes 2023-12-08T20:30:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 830915584 bytes 2023-12-08T20:30:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 869035008 bytes 2023-12-08T20:30:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 872057856 bytes 2023-12-08T20:30:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 899863552 bytes 2023-12-08T20:30:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 953755648 bytes 2023-12-08T20:30:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 991837184 bytes 2023-12-08T20:30:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1031703552 bytes 2023-12-08T20:30:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1113607168 bytes 2023-12-08T20:30:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1186004992 bytes 2023-12-08T20:30:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1199104000 bytes 2023-12-08T20:30:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1264037888 bytes 2023-12-08T20:30:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1306603520 bytes 2023-12-08T20:30:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1364538368 bytes 2023-12-08T20:30:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1427428352 bytes 2023-12-08T20:30:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1466084352 bytes 2023-12-08T20:30:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1492672512 bytes 2023-12-08T20:30:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1619591168 bytes 2023-12-08T20:30:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1681980416 bytes 2023-12-08T20:30:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1706580992 bytes 2023-12-08T20:30:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1761682432 bytes 2023-12-08T20:30:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1762297856 bytes 2023-12-08T20:30:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1805298688 bytes 2023-12-08T20:30:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1836545024 bytes 2023-12-08T20:30:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1883408384 bytes 2023-12-08T20:30:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1915531264 bytes 2023-12-08T20:30:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1928827904 bytes 2023-12-08T20:30:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2054623232 bytes 2023-12-08T20:30:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2089170944 bytes 2023-12-08T20:30:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2123294720 bytes 2023-12-08T20:30:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2199901184 bytes 2023-12-08T20:30:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:30:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:30:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:30:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:30:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:30:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:30:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:30:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:30:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:30:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:30:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:30:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:31:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:31:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:31:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:31:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:31:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:31:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:31:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533464064.0,"last_snapshot_bytes":532324352,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067386,"remote_snapshot_timestamp":1702067386,"replay_state":"idle"} 2023-12-08T20:31:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:31:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:30:37 2023-12-08T20:31:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:31:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:31:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:31:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:31:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:30:37 2023-12-08T20:31:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:31:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533464064.0,"last_snapshot_bytes":532324352,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067386,"remote_snapshot_timestamp":1702067386,"replay_state":"idle"} 2023-12-08T20:31:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:31:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:31:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:31:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:31:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:31:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:30:37 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:30:37 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:30:37 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:31:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:31:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:31:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:31:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:31:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:31:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:30:37 2023-12-08T20:31:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:31:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:31:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:31:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:31:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:31:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap7 2023-12-08T20:31:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap7 2023-12-08T20:31:00.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap7 2023-12-08T20:31:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:31:08.445 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-08T20:31:08.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-08T20:31:08.446 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-08T20:31:08.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:31:08.439+0000 7fb6fccd4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 34293) UID: 1000 2023-12-08T20:31:08.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:31:12.447 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-08T20:31:12.447 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:31:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:31:12.447 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-08T20:31:12.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-08T20:31:12.448 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-08T20:31:12.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-08T20:31:12.451 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-12-08T20:31:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:31:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:31:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:31:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:31:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1+ 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:31:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:31:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:31:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:31:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:31:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:31:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:31:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:31:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:31:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:31:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:31:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:31:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:31:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:31:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:31:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:31:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:31:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:31:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:31:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:31:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:31:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:31:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:31:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:31:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:31:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:31:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:31:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:31:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:31:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:31:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:31:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:31:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:31:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:31:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:31:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:31:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:31:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:31:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:31:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 29 2023-12-08T20:31:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:31:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:31:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()]+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:31:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:31:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:31:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=29 2023-12-08T20:31:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:31:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:31:32.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9187441878456583) > weight (0.25) 2023-12-08T20:31:32.329 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-08T20:31:32.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-08T20:31:32.330 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-08T20:31:32.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T20:31:32.323+0000 7f1eb18b2700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 34313) UID: 1000 2023-12-08T20:31:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:31:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:31:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:31:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:31:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:31:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:31:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:31:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:31:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 29 = 2023-12-08T20:31:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:31:32.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:31:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:31:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:31:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:31:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:31:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:31:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:31:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:31:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:31:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 29 = 2023-12-08T20:31:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:31:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:31:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:31:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:31:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:31:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:31:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:31:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 29 = 2023-12-08T20:31:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:31:34.331 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-08T20:31:34.331 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:31:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:31:34.332 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-08T20:31:34.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-08T20:31:34.332 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-08T20:31:34.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-08T20:31:34.335 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-08T20:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:31:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:31:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=29 2023-12-08T20:31:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 29 = 29 2023-12-08T20:31:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-08T20:31:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap7 2023-12-08T20:31:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:31:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:31:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 14 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:31:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:30:48 2023 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:31:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap6-clean ] 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap6-clean 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:31:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:31:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:31:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap7 2023-12-08T20:31:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap7 2023-12-08T20:31:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:31:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:31:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:31:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:31:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:31:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:31:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap7-clean 2023-12-08T20:31:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-08T20:31:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap7-clean 2023-12-08T20:31:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:31:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:31:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:31:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=6 2023-12-08T20:31:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:31:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:31:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 63850496 bytes 2023-12-08T20:31:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 125146112 bytes 2023-12-08T20:31:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 189686784 bytes 2023-12-08T20:31:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 248290304 bytes 2023-12-08T20:31:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 345863168 bytes 2023-12-08T20:31:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 364798976 bytes 2023-12-08T20:31:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 398913536 bytes 2023-12-08T20:31:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 404379648 bytes 2023-12-08T20:31:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 425745408 bytes 2023-12-08T20:31:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 471066624 bytes 2023-12-08T20:31:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 515619840 bytes 2023-12-08T20:31:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 602399744 bytes 2023-12-08T20:31:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 634770432 bytes 2023-12-08T20:31:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 664117248 bytes 2023-12-08T20:31:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 712424448 bytes 2023-12-08T20:31:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 737518592 bytes 2023-12-08T20:31:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 792144896 bytes 2023-12-08T20:31:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 861059072 bytes 2023-12-08T20:31:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 967851008 bytes 2023-12-08T20:31:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1041686528 bytes 2023-12-08T20:31:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1107733504 bytes 2023-12-08T20:31:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1194617856 bytes 2023-12-08T20:31:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1238305792 bytes 2023-12-08T20:31:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1270584320 bytes 2023-12-08T20:31:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1287311360 bytes 2023-12-08T20:31:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1367011328 bytes 2023-12-08T20:31:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1458428928 bytes 2023-12-08T20:31:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1459028992 bytes 2023-12-08T20:31:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1525209088 bytes 2023-12-08T20:31:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1564827648 bytes 2023-12-08T20:31:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1629840384 bytes 2023-12-08T20:31:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1656444928 bytes 2023-12-08T20:31:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1710239744 bytes 2023-12-08T20:31:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1765805056 bytes 2023-12-08T20:31:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1807156224 bytes 2023-12-08T20:31:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1828310016 bytes 2023-12-08T20:31:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1872104448 bytes 2023-12-08T20:31:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1919273984 bytes 2023-12-08T20:31:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:31:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:31:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:31:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:31:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:31:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:31:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:31:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:31:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":17366186.67,"bytes_per_snapshot":528579072.0,"last_snapshot_bytes":520985600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067491,"remote_snapshot_timestamp":1702067491,"replay_state":"idle"} 2023-12-08T20:31:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:31:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:31:37 2023-12-08T20:31:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:31:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:31:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:31:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:31:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:31:37 2023-12-08T20:31:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:31:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":17366186.67,"bytes_per_snapshot":528579072.0,"last_snapshot_bytes":520985600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067491,"remote_snapshot_timestamp":1702067491,"replay_state":"idle"} 2023-12-08T20:31:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:31:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:31:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:31:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:31:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:31:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:31:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:31:37 2023-12-08T20:31:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:31:37 2023-12-08T20:31:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:31:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:31:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:31:37 2023-12-08T20:31:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:31:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:31:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:31:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:31:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:31:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:31:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:31:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:31:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:31:37 2023-12-08T20:31:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:31:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:31:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:31:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:31:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:31:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:31:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap8 2023-12-08T20:31:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap8 2023-12-08T20:31:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:31:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:31:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:31:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap8 2023-12-08T20:31:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:32:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:32:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:32:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:32:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:32:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:32:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:32:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:32:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:32:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:32:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:32:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:32:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:32:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:32:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:32:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:32:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:32:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:32:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:32:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:32:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:32:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:32:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:32:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:32:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:32:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:32:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:32:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:32:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:32:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:32:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:32:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:32:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:32:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:32:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:32:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:32:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:32:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:32:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:32:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:32:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:32:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:32:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + echo cluster1-mirror-test.status 2023-12-08T20:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:sed -e s|[/ ]|_|g 2023-12-08T20:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:32:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 33 2023-12-08T20:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:32:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:32:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:32:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:32:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=33 2023-12-08T20:32:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:32:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:32:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:32:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:32:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:32:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:32:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:32:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:32:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:32:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:32:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 33 = 2023-12-08T20:32:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:32:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:32:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:32:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:32:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:32:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:32:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:32:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:32:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:32:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 33 = 2023-12-08T20:32:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:32:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:32:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:32:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:32:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:32:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:32:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:32:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:32:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:32:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 33 = 2023-12-08T20:32:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:32:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:32:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:32:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:32:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=33 2023-12-08T20:32:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 33 = 33 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap8 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:32:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:32:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-08T20:32:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 16 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:32:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:30:48 2023 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:32:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:32:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap7-clean ] 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap7-clean 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:32:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:32:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:32:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap8 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap8 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:32:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:32:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:32:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:32:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap8-clean 2023-12-08T20:32:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-08T20:32:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:32:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap8-clean 2023-12-08T20:32:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:32:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:32:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:32:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:32:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:32:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:32:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:32:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 41661440 bytes 2023-12-08T20:32:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 106848256 bytes 2023-12-08T20:32:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 192344064 bytes 2023-12-08T20:32:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 223280128 bytes 2023-12-08T20:32:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 249924608 bytes 2023-12-08T20:32:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 295112704 bytes 2023-12-08T20:32:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 323345408 bytes 2023-12-08T20:32:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 339069952 bytes 2023-12-08T20:32:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 402136064 bytes 2023-12-08T20:32:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 445028352 bytes 2023-12-08T20:32:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 482886656 bytes 2023-12-08T20:32:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 588523520 bytes 2023-12-08T20:32:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 685989888 bytes 2023-12-08T20:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 742733824 bytes 2023-12-08T20:32:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 765454336 bytes 2023-12-08T20:32:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 784092160 bytes 2023-12-08T20:32:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 867844096 bytes 2023-12-08T20:32:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 963785728 bytes 2023-12-08T20:32:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 1008423936 bytes 2023-12-08T20:32:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1053871104 bytes 2023-12-08T20:32:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1079745536 bytes 2023-12-08T20:32:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1092593664 bytes 2023-12-08T20:32:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1116307456 bytes 2023-12-08T20:32:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1142633472 bytes 2023-12-08T20:32:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1163360256 bytes 2023-12-08T20:32:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1182997504 bytes 2023-12-08T20:32:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1183620096 bytes 2023-12-08T20:32:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1256353792 bytes 2023-12-08T20:32:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1278860288 bytes 2023-12-08T20:32:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1329619968 bytes 2023-12-08T20:32:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1382057984 bytes 2023-12-08T20:32:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1390483456 bytes 2023-12-08T20:32:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1391064064 bytes 2023-12-08T20:32:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1483392000 bytes 2023-12-08T20:32:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:32:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:32:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:32:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":17366186.67,"bytes_per_snapshot":528579072.0,"last_snapshot_bytes":520985600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067491,"remote_snapshot_timestamp":1702067491,"replay_state":"idle"} 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:32:07 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:32:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:32:07 2023-12-08T20:32:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:32:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":17366186.67,"bytes_per_snapshot":528579072.0,"last_snapshot_bytes":520985600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067491,"remote_snapshot_timestamp":1702067491,"replay_state":"idle"} 2023-12-08T20:32:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:32:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:32:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:32:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:32:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:32:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:32:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:32:07 2023-12-08T20:32:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:32:07 2023-12-08T20:32:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:32:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:32:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:32:07 2023-12-08T20:32:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:32:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:32:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:32:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:32:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:32:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:32:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:32:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:32:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:32:07 2023-12-08T20:32:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:32:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:32:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:32:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:32:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap9 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap9 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap9 2023-12-08T20:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:32:53.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5317904084145862) > weight (0.25) 2023-12-08T20:32:53.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.295924348836171) > weight (0.25) 2023-12-08T20:32:53.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.625006330527449) > weight (0.25) 2023-12-08T20:32:53.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9285851995159138) > weight (0.25) 2023-12-08T20:32:53.453 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-08T20:33:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:33:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:33:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:33:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:33:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:33:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:33:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:33:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:33:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:33:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ localgrep cluster=cluster1 2023-12-08T20:33:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: -i state 2023-12-08T20:33:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:33:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:33:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:33:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:33:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:33:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:33:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:33:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:33:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:33:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:33:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:33:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:33:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:33:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:33:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:33:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:33:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:33:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:33:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:33:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:33:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:33:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:33:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:33:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:33:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:33:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:33:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep+ -i Replaying 2023-12-08T20:33:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:echo "state": "Replaying" 2023-12-08T20:33:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:33:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:33:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:33:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:33:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:33:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:33:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:33:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:33:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:33:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:33:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:33:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:33:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 37 2023-12-08T20:33:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:33:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:33:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:33:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:33:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=37 2023-12-08T20:33:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:33:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:33:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:33:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:33:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 37 = 2023-12-08T20:33:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:33:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:33:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:33:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:33:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:33:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 37 = 2023-12-08T20:33:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:33:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:33:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:33:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:33:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:33:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 37 = 2023-12-08T20:33:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:33:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=37 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 37 = 37 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap9 2023-12-08T20:33:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:33:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:33:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 18 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:33:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:32:25 2023 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:33:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:33:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap8-clean ] 2023-12-08T20:33:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-08T20:33:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap8-clean 2023-12-08T20:33:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:33:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:33:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:33:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:33:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:33:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap9 2023-12-08T20:33:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap9 2023-12-08T20:33:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:33:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:33:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:33:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:33:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:33:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap9-clean 2023-12-08T20:33:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-08T20:33:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap9-clean 2023-12-08T20:33:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:33:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror test 2023-12-08T20:33:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:33:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:33:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:33:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-08T20:33:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 44957696 bytes 2023-12-08T20:33:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 96396288 bytes 2023-12-08T20:33:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 176731136 bytes 2023-12-08T20:33:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 240131072 bytes 2023-12-08T20:33:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 270042112 bytes 2023-12-08T20:33:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 296613888 bytes 2023-12-08T20:33:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 329248768 bytes 2023-12-08T20:33:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 360996864 bytes 2023-12-08T20:33:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 409480192 bytes 2023-12-08T20:33:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 433333248 bytes 2023-12-08T20:33:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 491899904 bytes 2023-12-08T20:33:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 515609600 bytes 2023-12-08T20:33:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 558903296 bytes 2023-12-08T20:33:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 621336576 bytes 2023-12-08T20:33:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 640017408 bytes 2023-12-08T20:33:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 714394624 bytes 2023-12-08T20:33:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 774530048 bytes 2023-12-08T20:33:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 788023296 bytes 2023-12-08T20:33:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 821069824 bytes 2023-12-08T20:33:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 844638208 bytes 2023-12-08T20:33:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 944886784 bytes 2023-12-08T20:33:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 991785984 bytes 2023-12-08T20:33:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1060489216 bytes 2023-12-08T20:33:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1171535872 bytes 2023-12-08T20:33:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1228551168 bytes 2023-12-08T20:33:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1271281664 bytes 2023-12-08T20:33:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1314318336 bytes 2023-12-08T20:33:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1363255296 bytes 2023-12-08T20:33:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1384018944 bytes 2023-12-08T20:33:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1403715584 bytes 2023-12-08T20:33:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1464208384 bytes 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:33:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:33:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:33:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:33:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-08T20:33:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:33:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local description_pattern= 2023-12-08T20:33:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local service_pattern= 2023-12-08T20:33:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-08T20:33:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-08T20:33:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-08T20:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:test: 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: global_id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+replaying 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":514032128.0,"last_snapshot_bytes":507078656,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067538,"remote_snapshot_timestamp":1702067538,"replay_state":"idle"} 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: service: mirror.1 on smithi158 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:33:07 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: peer_sites: 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: name: cluster2 2023-12-08T20:33:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state: up+stopped 2023-12-08T20:33:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: description: local image is primary 2023-12-08T20:33:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: last_update: 2023-12-08 20:33:07 2023-12-08T20:33:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:33:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: state: up+replaying 2023-12-08T20:33:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^ description: .* /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:33:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":514032128.0,"last_snapshot_bytes":507078656,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702067538,"remote_snapshot_timestamp":1702067538,"replay_state":"idle"} 2023-12-08T20:33:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:33:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:33:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep ^up+ 2023-12-08T20:33:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:33:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep service: *mirror..* on /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:33:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: service: mirror.1 on smithi158 2023-12-08T20:33:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.mirror_status 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update=2023-12-08 20:33:07 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-08 20:33:07 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state_pattern=up+replaying 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local prev_last_update=2023-12-08 20:33:07 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror.mirror_status 2023-12-08T20:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:33:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror.mirror_status 2023-12-08T20:33:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-08T20:33:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:33:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_update state 2023-12-08T20:33:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-08T20:33:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ last_update=2023-12-08 20:33:07 2023-12-08T20:33:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-08T20:33:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=up+replaying 2023-12-08T20:33:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo up+replaying 2023-12-08T20:33:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep up+replaying 2023-12-08T20:33:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:up+replaying 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap10 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap10 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap10 2023-12-08T20:33:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:33:31.335 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-08T20:33:31.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-08T20:33:31.336 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-08T20:33:31.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:2023-12-08T20:33:31.330+0000 7f67d6aaa700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 34302) UID: 1000 2023-12-08T20:33:31.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:33:32.337 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-08T20:33:32.337 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:33:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:33:32.338 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-08T20:33:32.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-08T20:33:32.338 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-08T20:33:32.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-08T20:33:32.342 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-08T20:33:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:33:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:33:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:33:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local instance=cluster1 2023-12-08T20:33:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:33:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:33:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:33:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:33:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:33:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:33:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:33:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:33:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:33:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:33:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:33:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:33:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:33:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:33:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:33:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:33:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:33:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:33:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:33:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:33:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:33:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:33:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:33:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:33:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:33:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:33:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:33:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:33:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:33:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:33:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:33:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:33:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:33:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:33:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:33:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:33:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:33:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:33:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:33:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:33:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 41 2023-12-08T20:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:33:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:33:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=41 2023-12-08T20:33:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:33:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:33:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:33:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:33:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:33:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 41 = 2023-12-08T20:33:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:33:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:33:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:33:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:33:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:33:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 41 = 2023-12-08T20:33:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:33:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:33:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:33:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:33:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:33:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:33:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 41 = 2023-12-08T20:33:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:33:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:33:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=41 2023-12-08T20:33:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 41 = 41 2023-12-08T20:33:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-08T20:33:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:33:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap10 2023-12-08T20:33:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:33:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:33:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-08T20:33:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:33:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:33:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:33:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 20 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:32:25 2023 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:33:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:33:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snap9-clean ] 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap9-clean 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:33:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:33:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:33:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:33:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap10 2023-12-08T20:33:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:33:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:33:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap10 2023-12-08T20:33:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:33:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:33:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:33:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ clean_snap_name=snap10-clean 2023-12-08T20:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-08T20:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap10-clean 2023-12-08T20:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-08T20:34:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:34:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:34:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-08T20:34:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i+ statelocal 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: cluster=cluster1 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:34:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:34:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:34:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:34:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-08T20:34:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:34:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:34:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:34:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:34:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:34:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:34:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:34:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-08T20:34:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-08T20:34:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd mirror status : get status for rbd mirror 2023-12-08T20:34:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:admin_socket: invalid command 2023-12-08T20:34:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-08T20:34:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:1 2023-12-08T20:34:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:34:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:34:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-08T20:34:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:1 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 1 = cluster1:1 ] 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=1 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=1 2023-12-08T20:34:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-08T20:34:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:34:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:34:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-08T20:34:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-08T20:34:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:34:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:34:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:34:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-test.status 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-test.status 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:34:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:34:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-test.status 2023-12-08T20:34:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-test.status 2023-12-08T20:34:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:34:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:34:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-08T20:34:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-08T20:34:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 44 2023-12-08T20:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.status 2023-12-08T20:34:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-08T20:34:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:34:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:34:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=44 2023-12-08T20:34:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:34:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:34:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:34:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:34:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:34:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:34:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:34:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:34:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:34:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:34:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:34:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:34:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 44 = 2023-12-08T20:34:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:34:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:34:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.status 2023-12-08T20:34:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-08T20:34:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:34:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:34:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=44 2023-12-08T20:34:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 44 = 44 2023-12-08T20:34:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-08T20:34:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:34:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap10-clean 2023-12-08T20:34:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:34:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:34:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-08T20:34:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'test': 2023-12-08T20:34:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 512 MiB in 128 objects 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 21 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109841839e5 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109841839e5 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:33:58 2023 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:25:39 2023 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:34:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0bc9151c-4c05-41cd-9dce-df3edc7e54cd 2023-12-08T20:34:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:34:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 10 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap1 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap1 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap1 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap1-clean 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap1-clean 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:10.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.749230525717689) > weight (0.25) 2023-12-08T20:34:10.454 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-08T20:34:10.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-08T20:34:10.455 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-08T20:34:10.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:34:10.449+0000 7f5fa3f59700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 34295) UID: 1000 2023-12-08T20:34:10.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:34:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap2 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap2 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap2 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap2-clean 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap2-clean 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap3 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap3 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap3 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:15.455 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-08T20:34:15.455 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:34:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:34:15.456 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-08T20:34:15.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-08T20:34:15.456 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-08T20:34:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-08T20:34:15.459 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-08T20:34:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap3-clean 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap3-clean 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap4 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap4 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap4 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap4-clean 2023-12-08T20:34:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-08T20:34:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap4-clean 2023-12-08T20:34:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap5 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap5 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap5 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap5-clean 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap5-clean 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap6 2023-12-08T20:34:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap6 2023-12-08T20:34:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap6 2023-12-08T20:34:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap6-clean 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap6-clean 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap7 2023-12-08T20:34:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap7 2023-12-08T20:34:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap7 2023-12-08T20:34:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap7-clean 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap7-clean 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap8 2023-12-08T20:34:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap8 2023-12-08T20:34:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap8 2023-12-08T20:34:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap8-clean 2023-12-08T20:34:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-08T20:34:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap8-clean 2023-12-08T20:34:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap9 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap9 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap9 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap9-clean 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap9-clean 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap10 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap10 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap10 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap10-clean 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap10-clean 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export 2023-12-08T20:34:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-08T20:34:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-test.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-test.export 2023-12-08T20:34:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:34:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 10 2023-12-08T20:34:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap1 2023-12-08T20:34:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-08T20:34:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap1 2023-12-08T20:34:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-08T20:34:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap1-clean 2023-12-08T20:34:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-08T20:34:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap1-clean 2023-12-08T20:34:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-08T20:34:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap2 2023-12-08T20:34:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-08T20:34:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap2 2023-12-08T20:34:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-08T20:34:50.343 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-08T20:34:50.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-08T20:34:50.343 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-08T20:34:50.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:2023-12-08T20:34:50.337+0000 7fa93f60f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40697) UID: 1000 2023-12-08T20:34:50.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:34:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap2-clean 2023-12-08T20:34:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-08T20:34:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap2-clean 2023-12-08T20:34:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-08T20:34:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap3 2023-12-08T20:34:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-08T20:34:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap3 2023-12-08T20:34:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-08T20:34:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap3-clean 2023-12-08T20:34:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-08T20:34:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap3-clean 2023-12-08T20:34:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-08T20:34:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap4 2023-12-08T20:34:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-08T20:34:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap4 2023-12-08T20:34:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-08T20:34:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap4-clean 2023-12-08T20:34:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-08T20:34:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap4-clean 2023-12-08T20:34:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-08T20:34:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap5 2023-12-08T20:34:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-08T20:34:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap5 2023-12-08T20:34:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-08T20:34:56.344 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-08T20:34:56.344 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:34:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:34:56.345 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-08T20:34:56.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-08T20:34:56.345 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-08T20:34:56.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-08T20:34:56.347 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-08T20:34:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap5-clean 2023-12-08T20:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-08T20:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap5-clean 2023-12-08T20:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-08T20:34:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap6 2023-12-08T20:34:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-08T20:34:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap6 2023-12-08T20:34:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-08T20:34:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:34:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap6-clean 2023-12-08T20:34:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-08T20:34:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:34:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:34:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:34:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap6-clean 2023-12-08T20:34:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-08T20:35:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap7 2023-12-08T20:35:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-08T20:35:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap7 2023-12-08T20:35:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-08T20:35:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap7-clean 2023-12-08T20:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-08T20:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap7-clean 2023-12-08T20:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-08T20:35:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap8 2023-12-08T20:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-08T20:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap8 2023-12-08T20:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-08T20:35:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap8-clean 2023-12-08T20:35:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-08T20:35:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap8-clean 2023-12-08T20:35:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-08T20:35:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap9 2023-12-08T20:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-08T20:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap9 2023-12-08T20:35:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-08T20:35:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap9-clean 2023-12-08T20:35:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-08T20:35:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap9-clean 2023-12-08T20:35:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-08T20:35:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap10 2023-12-08T20:35:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-08T20:35:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap10 2023-12-08T20:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-08T20:35:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap10-clean 2023-12-08T20:35:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-08T20:35:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap=snap10-clean 2023-12-08T20:35:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-08T20:35:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing snap: 100% complete...done. 2023-12-08T20:35:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_image_retry cluster2 mirror test 2023-12-08T20:35:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ remove_image cluster2 mirror test 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-08T20:35:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-08T20:35:08.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T20:35:08.252+0000 7f07ce7ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5590d31bc000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-08T20:35:08.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T20:35:08.252+0000 7f07d4fd7700 -1 rbd::mirror::ImageReplayer: 0x5590cf1e8c80 [3/0bc9151c-4c05-41cd-9dce-df3edc7e54cd] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-08T20:35:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-08T20:35:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=deleted 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image_id= 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_image_id cluster1 mirror test 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-08T20:35:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-08T20:35:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-08T20:35:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image_id= 2023-12-08T20:35:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.1 2023-12-08T20:35:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_present cluster1 mirror test deleted 2023-12-08T20:35:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:35:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=deleted 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image_id= 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=deleted 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_image_id 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_image_id cluster1 mirror test 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=test 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-08T20:35:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-08T20:35:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-08T20:35:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_image_id= 2023-12-08T20:35:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -n 2023-12-08T20:35:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test deleted = deleted 2023-12-08T20:35:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:35:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ testlog TEST: create many images 2023-12-08T20:35:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee -a /tmp/tmp.b8SaxlVk4h/rbd-mirror.test.log 2023-12-08T20:35:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ date +%F %T 2023-12-08T20:35:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo 2023-12-08 20:35:10 TEST: create many images 2023-12-08T20:35:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:2023-12-08 20:35:10 TEST: create many images 2023-12-08T20:35:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ snap_name=snap 2023-12-08T20:35:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 50 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_1 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:35:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-08T20:35:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-08T20:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-08T20:35:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:35:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_1': 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12544f03a309 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12544f03a309 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:35:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:10 2023 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:10 2023 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:10 2023 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 52a3f781-3d39-4a12-bafb-6bc21cbfa607 2023-12-08T20:35:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:35:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:35:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-08T20:35:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:35:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:35:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:35:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:35:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:35:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 18312192 bytes 2023-12-08T20:35:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 71242752 bytes 2023-12-08T20:35:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 209838080 bytes 2023-12-08T20:35:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 250357760 bytes 2023-12-08T20:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 250969088 bytes 2023-12-08T20:35:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 305045504 bytes 2023-12-08T20:35:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 368121856 bytes 2023-12-08T20:35:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 430745600 bytes 2023-12-08T20:35:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 474734592 bytes 2023-12-08T20:35:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 490430464 bytes 2023-12-08T20:35:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 500937728 bytes 2023-12-08T20:35:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 545923072 bytes 2023-12-08T20:35:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 589102080 bytes 2023-12-08T20:35:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 613555200 bytes 2023-12-08T20:35:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 614167552 bytes 2023-12-08T20:35:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 616723456 bytes 2023-12-08T20:35:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 641930240 bytes 2023-12-08T20:35:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 664163328 bytes 2023-12-08T20:35:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 770728960 bytes 2023-12-08T20:35:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 807137280 bytes 2023-12-08T20:35:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 842129408 bytes 2023-12-08T20:35:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 897221632 bytes 2023-12-08T20:35:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 984459264 bytes 2023-12-08T20:35:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1007430656 bytes 2023-12-08T20:35:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1035348992 bytes 2023-12-08T20:35:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1057781760 bytes 2023-12-08T20:35:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1144366080 bytes 2023-12-08T20:35:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1191173120 bytes 2023-12-08T20:35:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1256498176 bytes 2023-12-08T20:35:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1257133056 bytes 2023-12-08T20:35:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1276134400 bytes 2023-12-08T20:35:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1358705664 bytes 2023-12-08T20:35:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1435778048 bytes 2023-12-08T20:35:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1476230144 bytes 2023-12-08T20:35:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1495192576 bytes 2023-12-08T20:35:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1520646144 bytes 2023-12-08T20:35:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:35:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:35:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:35:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:35:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_2 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:35:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:35:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:35:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:35:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-08T20:35:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:35:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-08T20:35:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:35:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-08T20:35:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:35:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_2': 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1260b7990b2d 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1260b7990b2d 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:35:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:16 2023 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:16 2023 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:16 2023 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 12c34aaf-ffa4-4588-bab2-a4673b2a7cee 2023-12-08T20:35:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:35:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:35:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-08T20:35:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:35:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:35:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=11 2023-12-08T20:35:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:35:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:35:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 65016832 bytes 2023-12-08T20:35:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 134944768 bytes 2023-12-08T20:35:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 168721408 bytes 2023-12-08T20:35:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 227527680 bytes 2023-12-08T20:35:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 261718016 bytes 2023-12-08T20:35:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 286152704 bytes 2023-12-08T20:35:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 328634368 bytes 2023-12-08T20:35:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 359385088 bytes 2023-12-08T20:35:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 382102528 bytes 2023-12-08T20:35:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 400691200 bytes 2023-12-08T20:35:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 436163584 bytes 2023-12-08T20:35:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 498209792 bytes 2023-12-08T20:35:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 521589760 bytes 2023-12-08T20:35:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 534833152 bytes 2023-12-08T20:35:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 548449280 bytes 2023-12-08T20:35:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 566327296 bytes 2023-12-08T20:35:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 577554432 bytes 2023-12-08T20:35:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 639895552 bytes 2023-12-08T20:35:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 679382016 bytes 2023-12-08T20:35:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 701030400 bytes 2023-12-08T20:35:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 707476480 bytes 2023-12-08T20:35:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 744688640 bytes 2023-12-08T20:35:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 770830336 bytes 2023-12-08T20:35:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 822551552 bytes 2023-12-08T20:35:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 887197696 bytes 2023-12-08T20:35:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 944684032 bytes 2023-12-08T20:35:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1024008192 bytes 2023-12-08T20:35:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1058219008 bytes 2023-12-08T20:35:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1235863552 bytes 2023-12-08T20:35:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1275452416 bytes 2023-12-08T20:35:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1305838592 bytes 2023-12-08T20:35:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1348240384 bytes 2023-12-08T20:35:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1385384960 bytes 2023-12-08T20:35:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1419572224 bytes 2023-12-08T20:35:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1428779008 bytes 2023-12-08T20:35:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1453342720 bytes 2023-12-08T20:35:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1474219008 bytes 2023-12-08T20:35:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1546397696 bytes 2023-12-08T20:35:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1605596160 bytes 2023-12-08T20:35:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1629718528 bytes 2023-12-08T20:35:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1660563456 bytes 2023-12-08T20:35:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1703700480 bytes 2023-12-08T20:35:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1763229696 bytes 2023-12-08T20:35:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1771127808 bytes 2023-12-08T20:35:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1800861696 bytes 2023-12-08T20:35:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1860178944 bytes 2023-12-08T20:35:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1868398592 bytes 2023-12-08T20:35:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1902648320 bytes 2023-12-08T20:35:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1946071040 bytes 2023-12-08T20:35:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 1998750720 bytes 2023-12-08T20:35:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2014902272 bytes 2023-12-08T20:35:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2038828032 bytes 2023-12-08T20:35:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2070421504 bytes 2023-12-08T20:35:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2105235456 bytes 2023-12-08T20:35:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2138924032 bytes 2023-12-08T20:35:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2141770752 bytes 2023-12-08T20:35:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2183858176 bytes 2023-12-08T20:35:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2231555072 bytes 2023-12-08T20:35:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2272552960 bytes 2023-12-08T20:35:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2337401856 bytes 2023-12-08T20:35:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2354290688 bytes 2023-12-08T20:35:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2390383616 bytes 2023-12-08T20:35:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2395322368 bytes 2023-12-08T20:35:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2403799040 bytes 2023-12-08T20:35:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2453093376 bytes 2023-12-08T20:35:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2519707648 bytes 2023-12-08T20:35:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2520324096 bytes 2023-12-08T20:35:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2579601408 bytes 2023-12-08T20:35:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 2622594048 bytes 2023-12-08T20:35:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 2626379776 bytes 2023-12-08T20:35:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 2668365824 bytes 2023-12-08T20:35:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 2696334336 bytes 2023-12-08T20:35:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 2741412864 bytes 2023-12-08T20:35:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 2798836736 bytes 2023-12-08T20:35:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 2803808256 bytes 2023-12-08T20:35:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 2865575936 bytes 2023-12-08T20:35:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 2904112128 bytes 2023-12-08T20:35:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 2945197056 bytes 2023-12-08T20:35:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 2991600640 bytes 2023-12-08T20:35:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3017969664 bytes 2023-12-08T20:35:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3057267712 bytes 2023-12-08T20:35:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3127999488 bytes 2023-12-08T20:35:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_3 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-08T20:35:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:35:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-08T20:35:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:35:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-08T20:35:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:35:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-08T20:35:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_3': 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 126c7113cac6 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.126c7113cac6 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:28 2023 2023-12-08T20:35:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:28 2023 2023-12-08T20:35:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:28 2023 2023-12-08T20:35:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:35:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:35:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 5d037cda-53a8-49bc-9a00-03b63f0f3612 2023-12-08T20:35:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:35:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:35:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-08T20:35:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:35:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:35:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:35:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:35:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:35:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 38652928 bytes 2023-12-08T20:35:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 95986688 bytes 2023-12-08T20:35:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 101003264 bytes 2023-12-08T20:35:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 152542208 bytes 2023-12-08T20:35:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 157423616 bytes 2023-12-08T20:35:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 202700800 bytes 2023-12-08T20:35:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 280640512 bytes 2023-12-08T20:35:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 348327936 bytes 2023-12-08T20:35:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 382731264 bytes 2023-12-08T20:35:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 455576576 bytes 2023-12-08T20:35:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 473625600 bytes 2023-12-08T20:35:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 537953280 bytes 2023-12-08T20:35:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 577143808 bytes 2023-12-08T20:35:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 636785664 bytes 2023-12-08T20:35:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 696378368 bytes 2023-12-08T20:35:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 782836736 bytes 2023-12-08T20:35:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 814750720 bytes 2023-12-08T20:35:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 887276544 bytes 2023-12-08T20:35:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 950862848 bytes 2023-12-08T20:35:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1013917696 bytes 2023-12-08T20:35:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1095194624 bytes 2023-12-08T20:35:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1163718656 bytes 2023-12-08T20:35:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1170187264 bytes 2023-12-08T20:35:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1229330432 bytes 2023-12-08T20:35:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1288812544 bytes 2023-12-08T20:35:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1319938048 bytes 2023-12-08T20:35:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1381502976 bytes 2023-12-08T20:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1386400768 bytes 2023-12-08T20:35:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1421480960 bytes 2023-12-08T20:35:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1429325824 bytes 2023-12-08T20:35:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1458241536 bytes 2023-12-08T20:35:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1476633600 bytes 2023-12-08T20:35:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1554900992 bytes 2023-12-08T20:35:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_4 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:35:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:35:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-08T20:35:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:35:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-08T20:35:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:35:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-08T20:35:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:35:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-08T20:35:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_4': 2023-12-08T20:35:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:35:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:35:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:35:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12782921d14 2023-12-08T20:35:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12782921d14 2023-12-08T20:35:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:34 2023 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:34 2023 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:34 2023 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: caca2379-4196-4a64-ad7b-2144d08a1309 2023-12-08T20:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-08T20:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:35:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:35:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:35:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:35:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 48150528 bytes 2023-12-08T20:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 69377024 bytes 2023-12-08T20:35:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 119718912 bytes 2023-12-08T20:35:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 217269248 bytes 2023-12-08T20:35:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 249650176 bytes 2023-12-08T20:35:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 364848128 bytes 2023-12-08T20:35:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 429853696 bytes 2023-12-08T20:35:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 456265728 bytes 2023-12-08T20:35:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 474780672 bytes 2023-12-08T20:35:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 508999680 bytes 2023-12-08T20:35:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 579210240 bytes 2023-12-08T20:35:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 620323840 bytes 2023-12-08T20:35:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 661910528 bytes 2023-12-08T20:35:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 703913984 bytes 2023-12-08T20:35:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 727593984 bytes 2023-12-08T20:35:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 791988224 bytes 2023-12-08T20:35:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 841300992 bytes 2023-12-08T20:35:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 889260032 bytes 2023-12-08T20:35:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 902325248 bytes 2023-12-08T20:35:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 959007744 bytes 2023-12-08T20:35:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1080233984 bytes 2023-12-08T20:35:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1091510272 bytes 2023-12-08T20:35:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1148592128 bytes 2023-12-08T20:35:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1212109824 bytes 2023-12-08T20:35:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1264161792 bytes 2023-12-08T20:35:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1302647808 bytes 2023-12-08T20:35:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1342042112 bytes 2023-12-08T20:35:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1342645248 bytes 2023-12-08T20:35:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1343243264 bytes 2023-12-08T20:35:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1371201536 bytes 2023-12-08T20:35:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1414291456 bytes 2023-12-08T20:35:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1441247232 bytes 2023-12-08T20:35:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1449699328 bytes 2023-12-08T20:35:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:35:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:35:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:35:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:35:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_5 2023-12-08T20:35:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-08T20:35:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:35:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:35:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:35:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:35:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-08T20:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-08T20:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:35:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-08T20:35:40.460 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7728253802913345) > weight (0.25) 2023-12-08T20:35:40.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9028731783789501) > weight (0.25) 2023-12-08T20:35:40.461 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-08T20:35:40.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-08T20:35:40.461 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-08T20:35:40.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:35:40.452+0000 7f94f099b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 34297) UID: 1000 2023-12-08T20:35:40.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:35:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:35:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-08T20:35:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_5': 2023-12-08T20:35:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:35:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:35:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1284fd958989 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1284fd958989 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:40 2023 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:40 2023 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:40 2023 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: ad893482-52ac-4c1f-95dc-b30b5b63a8dd 2023-12-08T20:35:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:35:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:35:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-08T20:35:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:35:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:35:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=10 2023-12-08T20:35:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:35:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:35:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 49873920 bytes 2023-12-08T20:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 67732480 bytes 2023-12-08T20:35:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 109715456 bytes 2023-12-08T20:35:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 142481408 bytes 2023-12-08T20:35:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 228796416 bytes 2023-12-08T20:35:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 254496768 bytes 2023-12-08T20:35:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 260715520 bytes 2023-12-08T20:35:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 280797184 bytes 2023-12-08T20:35:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 303676416 bytes 2023-12-08T20:35:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 304301056 bytes 2023-12-08T20:35:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 359359488 bytes 2023-12-08T20:35:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 394567680 bytes 2023-12-08T20:35:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 453310464 bytes 2023-12-08T20:35:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 468404224 bytes 2023-12-08T20:35:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 469013504 bytes 2023-12-08T20:35:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 509763584 bytes 2023-12-08T20:35:43.462 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-08T20:35:43.462 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:35:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:35:43.462 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-08T20:35:43.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-08T20:35:43.463 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-08T20:35:43.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-08T20:35:43.465 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-12-08T20:35:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 546993152 bytes 2023-12-08T20:35:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 547629056 bytes 2023-12-08T20:35:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 637509632 bytes 2023-12-08T20:35:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 647459840 bytes 2023-12-08T20:35:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 690366464 bytes 2023-12-08T20:35:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 708070400 bytes 2023-12-08T20:35:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 708657152 bytes 2023-12-08T20:35:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 709278720 bytes 2023-12-08T20:35:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 776128512 bytes 2023-12-08T20:35:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 822094848 bytes 2023-12-08T20:35:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 878799872 bytes 2023-12-08T20:35:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 904539136 bytes 2023-12-08T20:35:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1041012736 bytes 2023-12-08T20:35:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1071561728 bytes 2023-12-08T20:35:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1129930752 bytes 2023-12-08T20:35:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1130528768 bytes 2023-12-08T20:35:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1165602816 bytes 2023-12-08T20:35:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1189240832 bytes 2023-12-08T20:35:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1196384256 bytes 2023-12-08T20:35:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1227790336 bytes 2023-12-08T20:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1259305984 bytes 2023-12-08T20:35:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1323827200 bytes 2023-12-08T20:35:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1346541568 bytes 2023-12-08T20:35:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1377269760 bytes 2023-12-08T20:35:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1402896384 bytes 2023-12-08T20:35:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1416131584 bytes 2023-12-08T20:35:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1464826880 bytes 2023-12-08T20:35:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1529292800 bytes 2023-12-08T20:35:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1551826944 bytes 2023-12-08T20:35:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1589741568 bytes 2023-12-08T20:35:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1613854720 bytes 2023-12-08T20:35:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1644601344 bytes 2023-12-08T20:35:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1657962496 bytes 2023-12-08T20:35:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 1697452032 bytes 2023-12-08T20:35:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 1707193344 bytes 2023-12-08T20:35:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 1740721152 bytes 2023-12-08T20:35:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 1818759168 bytes 2023-12-08T20:35:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 1835071488 bytes 2023-12-08T20:35:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 1898336256 bytes 2023-12-08T20:35:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 1905346560 bytes 2023-12-08T20:35:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 1940348928 bytes 2023-12-08T20:35:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 1989078016 bytes 2023-12-08T20:35:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2043725824 bytes 2023-12-08T20:35:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2044344320 bytes 2023-12-08T20:35:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2073424896 bytes 2023-12-08T20:35:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2094078976 bytes 2023-12-08T20:35:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2123621376 bytes 2023-12-08T20:35:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2181479424 bytes 2023-12-08T20:35:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2192967680 bytes 2023-12-08T20:35:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2232885248 bytes 2023-12-08T20:35:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2249568256 bytes 2023-12-08T20:35:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2312521728 bytes 2023-12-08T20:35:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 2343548928 bytes 2023-12-08T20:35:50.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 2412701696 bytes 2023-12-08T20:35:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 2464456704 bytes 2023-12-08T20:35:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 2558223360 bytes 2023-12-08T20:35:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 2599240704 bytes 2023-12-08T20:35:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 2647092224 bytes 2023-12-08T20:35:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 2715382784 bytes 2023-12-08T20:35:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 2760154112 bytes 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_6 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:35:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-08T20:35:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:35:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-08T20:35:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:35:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:35:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-08T20:35:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:35:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-08T20:35:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_6': 2023-12-08T20:35:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1294829eac1a 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1294829eac1a 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:51 2023 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:51 2023 2023-12-08T20:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:51 2023 2023-12-08T20:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: d6992038-914b-4ae5-92f9-4acfb9dca251 2023-12-08T20:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-08T20:35:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:35:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:35:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:35:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:35:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=10 2023-12-08T20:35:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:35:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:35:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 26590208 bytes 2023-12-08T20:35:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 83548160 bytes 2023-12-08T20:35:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 94055424 bytes 2023-12-08T20:35:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 125220864 bytes 2023-12-08T20:35:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 143631360 bytes 2023-12-08T20:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 160283648 bytes 2023-12-08T20:35:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 160902144 bytes 2023-12-08T20:35:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 218002432 bytes 2023-12-08T20:35:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 218614784 bytes 2023-12-08T20:35:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 253271040 bytes 2023-12-08T20:35:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 321269760 bytes 2023-12-08T20:35:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 336045056 bytes 2023-12-08T20:35:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 344750080 bytes 2023-12-08T20:35:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 410621952 bytes 2023-12-08T20:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 488231936 bytes 2023-12-08T20:35:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 573887488 bytes 2023-12-08T20:35:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 661523456 bytes 2023-12-08T20:35:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 686460928 bytes 2023-12-08T20:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 724297728 bytes 2023-12-08T20:35:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 759229440 bytes 2023-12-08T20:35:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 777027584 bytes 2023-12-08T20:35:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 811031552 bytes 2023-12-08T20:35:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 854282240 bytes 2023-12-08T20:35:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 928146432 bytes 2023-12-08T20:35:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 998533120 bytes 2023-12-08T20:35:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1037772800 bytes 2023-12-08T20:35:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1151778816 bytes 2023-12-08T20:35:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1240158208 bytes 2023-12-08T20:35:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1301344256 bytes 2023-12-08T20:35:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1358596096 bytes 2023-12-08T20:35:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1383301120 bytes 2023-12-08T20:35:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1398774784 bytes 2023-12-08T20:35:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1410709504 bytes 2023-12-08T20:35:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1433168896 bytes 2023-12-08T20:35:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1477471232 bytes 2023-12-08T20:35:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1519398912 bytes 2023-12-08T20:35:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1519997952 bytes 2023-12-08T20:35:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1549503488 bytes 2023-12-08T20:35:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1569425408 bytes 2023-12-08T20:35:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1613857792 bytes 2023-12-08T20:35:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1651782656 bytes 2023-12-08T20:35:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1701559296 bytes 2023-12-08T20:35:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1702166528 bytes 2023-12-08T20:35:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1746405376 bytes 2023-12-08T20:35:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1802018816 bytes 2023-12-08T20:35:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1831289856 bytes 2023-12-08T20:35:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1919995904 bytes 2023-12-08T20:35:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1940180992 bytes 2023-12-08T20:35:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1984256000 bytes 2023-12-08T20:35:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2021392384 bytes 2023-12-08T20:35:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2099914752 bytes 2023-12-08T20:35:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2141914112 bytes 2023-12-08T20:35:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2169035776 bytes 2023-12-08T20:35:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2246603776 bytes 2023-12-08T20:36:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2325720064 bytes 2023-12-08T20:36:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2387996672 bytes 2023-12-08T20:36:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2402734080 bytes 2023-12-08T20:36:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2491544576 bytes 2023-12-08T20:36:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2520275968 bytes 2023-12-08T20:36:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2593292288 bytes 2023-12-08T20:36:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2670607360 bytes 2023-12-08T20:36:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2756153344 bytes 2023-12-08T20:36:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2786544640 bytes 2023-12-08T20:36:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2859321344 bytes 2023-12-08T20:36:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2870978560 bytes 2023-12-08T20:36:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2953500672 bytes 2023-12-08T20:36:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2976332800 bytes 2023-12-08T20:36:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3010764800 bytes 2023-12-08T20:36:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:36:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:36:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:36:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_7 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-08T20:36:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:36:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-08T20:36:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:36:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-08T20:36:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:36:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-08T20:36:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_7': 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12a069ddff50 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12a069ddff50 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:02 2023 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:02 2023 2023-12-08T20:36:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:02 2023 2023-12-08T20:36:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:36:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:36:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 9345084e-0ac5-4beb-bc92-c8db39509c36 2023-12-08T20:36:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:36:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:36:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-08T20:36:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:36:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:36:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=12 2023-12-08T20:36:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:36:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:36:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 22406144 bytes 2023-12-08T20:36:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 54333440 bytes 2023-12-08T20:36:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 154285056 bytes 2023-12-08T20:36:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 199815168 bytes 2023-12-08T20:36:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 264775680 bytes 2023-12-08T20:36:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 295656448 bytes 2023-12-08T20:36:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 350101504 bytes 2023-12-08T20:36:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 385355776 bytes 2023-12-08T20:36:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 434636800 bytes 2023-12-08T20:36:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 496077824 bytes 2023-12-08T20:36:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 561114112 bytes 2023-12-08T20:36:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 568714240 bytes 2023-12-08T20:36:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 647992320 bytes 2023-12-08T20:36:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 657275904 bytes 2023-12-08T20:36:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 689312768 bytes 2023-12-08T20:36:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 709586944 bytes 2023-12-08T20:36:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 746180608 bytes 2023-12-08T20:36:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 808928256 bytes 2023-12-08T20:36:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 862050304 bytes 2023-12-08T20:36:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 882951168 bytes 2023-12-08T20:36:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 933113856 bytes 2023-12-08T20:36:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 980753408 bytes 2023-12-08T20:36:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1015138304 bytes 2023-12-08T20:36:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1068270592 bytes 2023-12-08T20:36:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1130321920 bytes 2023-12-08T20:36:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1181436928 bytes 2023-12-08T20:36:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1214446592 bytes 2023-12-08T20:36:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1262623744 bytes 2023-12-08T20:36:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1305150464 bytes 2023-12-08T20:36:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1341132800 bytes 2023-12-08T20:36:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1391191040 bytes 2023-12-08T20:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1468632064 bytes 2023-12-08T20:36:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1513036800 bytes 2023-12-08T20:36:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1530766336 bytes 2023-12-08T20:36:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1629384704 bytes 2023-12-08T20:36:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1698902016 bytes 2023-12-08T20:36:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1770509312 bytes 2023-12-08T20:36:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1850592256 bytes 2023-12-08T20:36:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1938935808 bytes 2023-12-08T20:36:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 2067354624 bytes 2023-12-08T20:36:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 2104226816 bytes 2023-12-08T20:36:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 2201282560 bytes 2023-12-08T20:36:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2272758784 bytes 2023-12-08T20:36:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2284093440 bytes 2023-12-08T20:36:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2306708480 bytes 2023-12-08T20:36:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2336091136 bytes 2023-12-08T20:36:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2394999808 bytes 2023-12-08T20:36:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2436113408 bytes 2023-12-08T20:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2502716416 bytes 2023-12-08T20:36:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2532703232 bytes 2023-12-08T20:36:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2593680384 bytes 2023-12-08T20:36:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2743565312 bytes 2023-12-08T20:36:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2797497344 bytes 2023-12-08T20:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2841675776 bytes 2023-12-08T20:36:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2888006656 bytes 2023-12-08T20:36:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2906774528 bytes 2023-12-08T20:36:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2959357952 bytes 2023-12-08T20:36:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2989076480 bytes 2023-12-08T20:36:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 3013203968 bytes 2023-12-08T20:36:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 3071824896 bytes 2023-12-08T20:36:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 3122403328 bytes 2023-12-08T20:36:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 3195722752 bytes 2023-12-08T20:36:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 3292054528 bytes 2023-12-08T20:36:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 3314370560 bytes 2023-12-08T20:36:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 3372780544 bytes 2023-12-08T20:36:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 3424833536 bytes 2023-12-08T20:36:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 3456668672 bytes 2023-12-08T20:36:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3488999424 bytes 2023-12-08T20:36:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3534853120 bytes 2023-12-08T20:36:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3599793152 bytes 2023-12-08T20:36:14.348 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7215447241815446) > weight (0.25) 2023-12-08T20:36:14.349 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-08T20:36:14.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-08T20:36:14.349 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-08T20:36:14.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T20:36:14.340+0000 7f07d57d8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39568) UID: 1000 2023-12-08T20:36:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3610215424 bytes 2023-12-08T20:36:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3655038976 bytes 2023-12-08T20:36:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3678829568 bytes 2023-12-08T20:36:14.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:36:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3707175936 bytes 2023-12-08T20:36:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3749389312 bytes 2023-12-08T20:36:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:36:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:36:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:36:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:36:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_8 2023-12-08T20:36:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:36:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:36:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:36:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-08T20:36:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:36:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-08T20:36:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:36:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-08T20:36:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:36:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_8': 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12acf6a25ad2 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12acf6a25ad2 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:36:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:15 2023 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:15 2023 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:15 2023 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: eec2c21a-48a4-4bfd-b1ce-523278c74bed 2023-12-08T20:36:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:36:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:36:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-08T20:36:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:36:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:36:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=14 2023-12-08T20:36:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:36:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:36:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 28852224 bytes 2023-12-08T20:36:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 45189120 bytes 2023-12-08T20:36:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 90384384 bytes 2023-12-08T20:36:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 120786944 bytes 2023-12-08T20:36:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 132418560 bytes 2023-12-08T20:36:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 180727808 bytes 2023-12-08T20:36:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 260720640 bytes 2023-12-08T20:36:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 308912128 bytes 2023-12-08T20:36:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 355836928 bytes 2023-12-08T20:36:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 455567360 bytes 2023-12-08T20:36:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 560864256 bytes 2023-12-08T20:36:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 586395648 bytes 2023-12-08T20:36:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 624619520 bytes 2023-12-08T20:36:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 638578688 bytes 2023-12-08T20:36:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 703307776 bytes 2023-12-08T20:36:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 744228864 bytes 2023-12-08T20:36:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 775086080 bytes 2023-12-08T20:36:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 789340160 bytes 2023-12-08T20:36:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 803290112 bytes 2023-12-08T20:36:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 856702976 bytes 2023-12-08T20:36:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 891196416 bytes 2023-12-08T20:36:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 950333440 bytes 2023-12-08T20:36:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 995100672 bytes 2023-12-08T20:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1079382016 bytes 2023-12-08T20:36:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1104701440 bytes 2023-12-08T20:36:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1156221952 bytes 2023-12-08T20:36:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1226232832 bytes 2023-12-08T20:36:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1275829248 bytes 2023-12-08T20:36:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1315297280 bytes 2023-12-08T20:36:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1343415296 bytes 2023-12-08T20:36:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1402308608 bytes 2023-12-08T20:36:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1462234112 bytes 2023-12-08T20:36:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1499013120 bytes 2023-12-08T20:36:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1499632640 bytes 2023-12-08T20:36:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1581118464 bytes 2023-12-08T20:36:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1620597760 bytes 2023-12-08T20:36:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1727828992 bytes 2023-12-08T20:36:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1728420864 bytes 2023-12-08T20:36:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1823284224 bytes 2023-12-08T20:36:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1853961216 bytes 2023-12-08T20:36:22.349 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-08T20:36:22.350 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:36:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:36:22.350 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-08T20:36:22.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-08T20:36:22.350 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-08T20:36:22.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-08T20:36:22.353 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2023-12-08T20:36:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1900985344 bytes 2023-12-08T20:36:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1915417600 bytes 2023-12-08T20:36:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1929391104 bytes 2023-12-08T20:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1960315904 bytes 2023-12-08T20:36:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1969661952 bytes 2023-12-08T20:36:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1976919040 bytes 2023-12-08T20:36:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2022548480 bytes 2023-12-08T20:36:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2044778496 bytes 2023-12-08T20:36:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2084134912 bytes 2023-12-08T20:36:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2084739072 bytes 2023-12-08T20:36:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2123254784 bytes 2023-12-08T20:36:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2180993024 bytes 2023-12-08T20:36:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2263463936 bytes 2023-12-08T20:36:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2352972800 bytes 2023-12-08T20:36:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2434080768 bytes 2023-12-08T20:36:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2480637952 bytes 2023-12-08T20:36:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2544776192 bytes 2023-12-08T20:36:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2587686912 bytes 2023-12-08T20:36:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2616193024 bytes 2023-12-08T20:36:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2630082560 bytes 2023-12-08T20:36:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2704792576 bytes 2023-12-08T20:36:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2726923264 bytes 2023-12-08T20:36:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2767068160 bytes 2023-12-08T20:36:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2786653184 bytes 2023-12-08T20:36:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2810056704 bytes 2023-12-08T20:36:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2892895232 bytes 2023-12-08T20:36:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2946169856 bytes 2023-12-08T20:36:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2964492288 bytes 2023-12-08T20:36:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3005872128 bytes 2023-12-08T20:36:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3038204928 bytes 2023-12-08T20:36:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3076005888 bytes 2023-12-08T20:36:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3097321472 bytes 2023-12-08T20:36:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3138786304 bytes 2023-12-08T20:36:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3165894656 bytes 2023-12-08T20:36:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3175988224 bytes 2023-12-08T20:36:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3245526016 bytes 2023-12-08T20:36:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3295525888 bytes 2023-12-08T20:36:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3356899328 bytes 2023-12-08T20:36:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3397141504 bytes 2023-12-08T20:36:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3441094656 bytes 2023-12-08T20:36:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3499823104 bytes 2023-12-08T20:36:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3538080768 bytes 2023-12-08T20:36:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3594589184 bytes 2023-12-08T20:36:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8400 IOs, 3595214848 bytes 2023-12-08T20:36:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8500 IOs, 3595834368 bytes 2023-12-08T20:36:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8600 IOs, 3619872768 bytes 2023-12-08T20:36:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8700 IOs, 3620497408 bytes 2023-12-08T20:36:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8800 IOs, 3670754304 bytes 2023-12-08T20:36:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8900 IOs, 3793511424 bytes 2023-12-08T20:36:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9000 IOs, 3810024448 bytes 2023-12-08T20:36:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9100 IOs, 3870185472 bytes 2023-12-08T20:36:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9200 IOs, 3920191488 bytes 2023-12-08T20:36:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:9300 IOs, 3920806912 bytes 2023-12-08T20:36:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:36:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:36:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:36:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:36:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_9 2023-12-08T20:36:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-08T20:36:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-08T20:36:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:36:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-08T20:36:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:36:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-08T20:36:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:36:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_9': 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12c29a6c46c2 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12c29a6c46c2 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:30 2023 2023-12-08T20:36:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:30 2023 2023-12-08T20:36:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:30 2023 2023-12-08T20:36:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:36:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:36:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 68475f85-0f1a-456b-ad04-ebe94e35d0a4 2023-12-08T20:36:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:36:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:36:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-08T20:36:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:36:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:36:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=13 2023-12-08T20:36:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:36:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:36:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 22536192 bytes 2023-12-08T20:36:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 62820352 bytes 2023-12-08T20:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 111511552 bytes 2023-12-08T20:36:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 161488896 bytes 2023-12-08T20:36:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 164804608 bytes 2023-12-08T20:36:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 171948032 bytes 2023-12-08T20:36:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 251934720 bytes 2023-12-08T20:36:32.429 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:36:32.416+0000 7fb534865700 -1 reset not still connected to 0x558a072d4f70 2023-12-08T20:36:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 291991552 bytes 2023-12-08T20:36:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 312796160 bytes 2023-12-08T20:36:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 315307008 bytes 2023-12-08T20:36:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 362109952 bytes 2023-12-08T20:36:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 414829568 bytes 2023-12-08T20:36:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 484882432 bytes 2023-12-08T20:36:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 529007616 bytes 2023-12-08T20:36:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 539546624 bytes 2023-12-08T20:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 607497216 bytes 2023-12-08T20:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 608100352 bytes 2023-12-08T20:36:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 652206080 bytes 2023-12-08T20:36:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 668692480 bytes 2023-12-08T20:36:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 710794240 bytes 2023-12-08T20:36:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 794564608 bytes 2023-12-08T20:36:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 835124224 bytes 2023-12-08T20:36:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 891121664 bytes 2023-12-08T20:36:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 927523840 bytes 2023-12-08T20:36:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 986319872 bytes 2023-12-08T20:36:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1008855040 bytes 2023-12-08T20:36:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1038401536 bytes 2023-12-08T20:36:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1049268224 bytes 2023-12-08T20:36:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1084014592 bytes 2023-12-08T20:36:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1157698560 bytes 2023-12-08T20:36:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1169086464 bytes 2023-12-08T20:36:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1214543872 bytes 2023-12-08T20:36:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1250102272 bytes 2023-12-08T20:36:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1268284416 bytes 2023-12-08T20:36:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1393852416 bytes 2023-12-08T20:36:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1470442496 bytes 2023-12-08T20:36:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1517215744 bytes 2023-12-08T20:36:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1531402240 bytes 2023-12-08T20:36:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1571470336 bytes 2023-12-08T20:36:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1622104064 bytes 2023-12-08T20:36:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1637363712 bytes 2023-12-08T20:36:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1707576320 bytes 2023-12-08T20:36:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1708180480 bytes 2023-12-08T20:36:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1762800640 bytes 2023-12-08T20:36:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1871961088 bytes 2023-12-08T20:36:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1907144704 bytes 2023-12-08T20:36:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1964802048 bytes 2023-12-08T20:36:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2026465280 bytes 2023-12-08T20:36:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2055222272 bytes 2023-12-08T20:36:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2121928704 bytes 2023-12-08T20:36:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2230988800 bytes 2023-12-08T20:36:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2274449408 bytes 2023-12-08T20:36:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2275083264 bytes 2023-12-08T20:36:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2294298624 bytes 2023-12-08T20:36:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2331002880 bytes 2023-12-08T20:36:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2370777088 bytes 2023-12-08T20:36:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2408301568 bytes 2023-12-08T20:36:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2465513472 bytes 2023-12-08T20:36:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2484268032 bytes 2023-12-08T20:36:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2588853248 bytes 2023-12-08T20:36:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2664059904 bytes 2023-12-08T20:36:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2686606336 bytes 2023-12-08T20:36:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2751956992 bytes 2023-12-08T20:36:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2800406528 bytes 2023-12-08T20:36:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2927775744 bytes 2023-12-08T20:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2935657472 bytes 2023-12-08T20:36:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2967189504 bytes 2023-12-08T20:36:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3036966912 bytes 2023-12-08T20:36:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3095687168 bytes 2023-12-08T20:36:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3098306560 bytes 2023-12-08T20:36:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3188389888 bytes 2023-12-08T20:36:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3213324288 bytes 2023-12-08T20:36:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3225940992 bytes 2023-12-08T20:36:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3273862144 bytes 2023-12-08T20:36:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3276934144 bytes 2023-12-08T20:36:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3312370688 bytes 2023-12-08T20:36:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3345464320 bytes 2023-12-08T20:36:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3420646400 bytes 2023-12-08T20:36:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3469046784 bytes 2023-12-08T20:36:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3534308352 bytes 2023-12-08T20:36:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3556419584 bytes 2023-12-08T20:36:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3622573056 bytes 2023-12-08T20:36:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3665556480 bytes 2023-12-08T20:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_10 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-08T20:36:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:36:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-08T20:36:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:36:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-08T20:36:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:36:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-08T20:36:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_10': 2023-12-08T20:36:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:36:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:36:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:36:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12ce8e8c3bef 2023-12-08T20:36:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12ce8e8c3bef 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:44 2023 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:44 2023 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:44 2023 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f5b9b8e7-d3c4-4866-97bf-4f179a9e81f1 2023-12-08T20:36:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:36:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:36:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-08T20:36:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:36:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:36:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=12 2023-12-08T20:36:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:36:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:36:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 54667264 bytes 2023-12-08T20:36:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 152459264 bytes 2023-12-08T20:36:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 234146816 bytes 2023-12-08T20:36:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 274973696 bytes 2023-12-08T20:36:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 360510464 bytes 2023-12-08T20:36:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 394793984 bytes 2023-12-08T20:36:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 443407360 bytes 2023-12-08T20:36:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 558424064 bytes 2023-12-08T20:36:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 654188544 bytes 2023-12-08T20:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 700637184 bytes 2023-12-08T20:36:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 729532416 bytes 2023-12-08T20:36:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 780564480 bytes 2023-12-08T20:36:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 788658176 bytes 2023-12-08T20:36:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 805982208 bytes 2023-12-08T20:36:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 849779712 bytes 2023-12-08T20:36:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 880755712 bytes 2023-12-08T20:36:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 917712896 bytes 2023-12-08T20:36:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 978386944 bytes 2023-12-08T20:36:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 1023841280 bytes 2023-12-08T20:36:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1048384512 bytes 2023-12-08T20:36:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1090627584 bytes 2023-12-08T20:36:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1118377984 bytes 2023-12-08T20:36:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1167224832 bytes 2023-12-08T20:36:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1284478976 bytes 2023-12-08T20:36:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1344236544 bytes 2023-12-08T20:36:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1432894464 bytes 2023-12-08T20:36:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1475133440 bytes 2023-12-08T20:36:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1552813056 bytes 2023-12-08T20:36:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1652189184 bytes 2023-12-08T20:36:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1687194624 bytes 2023-12-08T20:36:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1738860544 bytes 2023-12-08T20:36:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1790841856 bytes 2023-12-08T20:36:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1825503232 bytes 2023-12-08T20:36:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1921930240 bytes 2023-12-08T20:36:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1971013632 bytes 2023-12-08T20:36:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 2010110976 bytes 2023-12-08T20:36:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 2048319488 bytes 2023-12-08T20:36:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 2099935232 bytes 2023-12-08T20:36:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 2106143744 bytes 2023-12-08T20:36:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 2157423616 bytes 2023-12-08T20:36:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 2220824576 bytes 2023-12-08T20:36:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 2256582656 bytes 2023-12-08T20:36:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2311198720 bytes 2023-12-08T20:36:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2317289472 bytes 2023-12-08T20:36:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2335809536 bytes 2023-12-08T20:36:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2361219072 bytes 2023-12-08T20:36:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2425486336 bytes 2023-12-08T20:36:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2478095360 bytes 2023-12-08T20:36:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2523982848 bytes 2023-12-08T20:36:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2625533952 bytes 2023-12-08T20:36:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2714934272 bytes 2023-12-08T20:36:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2751708160 bytes 2023-12-08T20:36:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2833211392 bytes 2023-12-08T20:36:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2962616320 bytes 2023-12-08T20:36:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 3042863104 bytes 2023-12-08T20:36:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 3114993664 bytes 2023-12-08T20:36:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 3146449920 bytes 2023-12-08T20:36:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 3205359616 bytes 2023-12-08T20:36:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 3213600768 bytes 2023-12-08T20:36:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 3269443584 bytes 2023-12-08T20:36:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 3302894592 bytes 2023-12-08T20:36:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 3309931520 bytes 2023-12-08T20:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 3357546496 bytes 2023-12-08T20:36:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 3380926464 bytes 2023-12-08T20:36:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 3395248128 bytes 2023-12-08T20:36:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 3416808448 bytes 2023-12-08T20:36:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 3441287168 bytes 2023-12-08T20:36:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3495083008 bytes 2023-12-08T20:36:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3547668480 bytes 2023-12-08T20:36:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3577303040 bytes 2023-12-08T20:36:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3596827648 bytes 2023-12-08T20:36:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3597435904 bytes 2023-12-08T20:36:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3610997760 bytes 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_11 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:36:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2023-12-08T20:36:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:36:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-08T20:36:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:36:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:36:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-08T20:36:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:36:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-08T20:36:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_11': 2023-12-08T20:36:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:36:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:36:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:36:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12da4fc03d81 2023-12-08T20:36:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12da4fc03d81 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:57 2023 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:57 2023 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:57 2023 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 30b31997-991d-4daf-b0e8-576295786fc4 2023-12-08T20:36:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:36:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:36:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-08T20:36:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:36:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:36:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:36:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:36:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=6 2023-12-08T20:36:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:36:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:36:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 53906432 bytes 2023-12-08T20:36:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 117216256 bytes 2023-12-08T20:36:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 217424896 bytes 2023-12-08T20:36:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 343553024 bytes 2023-12-08T20:36:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 384943104 bytes 2023-12-08T20:36:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 438938624 bytes 2023-12-08T20:36:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 465498112 bytes 2023-12-08T20:36:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 507457536 bytes 2023-12-08T20:36:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 563023872 bytes 2023-12-08T20:37:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 613083136 bytes 2023-12-08T20:37:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 708749312 bytes 2023-12-08T20:37:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 751379456 bytes 2023-12-08T20:37:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 807819264 bytes 2023-12-08T20:37:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 870788096 bytes 2023-12-08T20:37:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 908329984 bytes 2023-12-08T20:37:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 991360000 bytes 2023-12-08T20:37:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 991978496 bytes 2023-12-08T20:37:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 1032939520 bytes 2023-12-08T20:37:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 1036086272 bytes 2023-12-08T20:37:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1067890688 bytes 2023-12-08T20:37:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1095418880 bytes 2023-12-08T20:37:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1107175424 bytes 2023-12-08T20:37:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1124845568 bytes 2023-12-08T20:37:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1205211136 bytes 2023-12-08T20:37:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1236238336 bytes 2023-12-08T20:37:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1266247680 bytes 2023-12-08T20:37:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1317612544 bytes 2023-12-08T20:37:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1389871104 bytes 2023-12-08T20:37:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1452237824 bytes 2023-12-08T20:37:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1530529792 bytes 2023-12-08T20:37:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1609436160 bytes 2023-12-08T20:37:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1649320960 bytes 2023-12-08T20:37:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1674380288 bytes 2023-12-08T20:37:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1706691584 bytes 2023-12-08T20:37:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1740365824 bytes 2023-12-08T20:37:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1813613568 bytes 2023-12-08T20:37:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_12 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:37:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2023-12-08T20:37:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:37:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-08T20:37:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-08T20:37:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:37:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_12': 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12e65fbdf195 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12e65fbdf195 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:04 2023 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:04 2023 2023-12-08T20:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:04 2023 2023-12-08T20:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 3c3d0826-2049-4f88-b8db-1a6dcc0c3b73 2023-12-08T20:37:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-08T20:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:37:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=7 2023-12-08T20:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:37:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:37:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 10089472 bytes 2023-12-08T20:37:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 24700928 bytes 2023-12-08T20:37:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 33741824 bytes 2023-12-08T20:37:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 43316224 bytes 2023-12-08T20:37:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 86238208 bytes 2023-12-08T20:37:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 102013952 bytes 2023-12-08T20:37:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 162527232 bytes 2023-12-08T20:37:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 186102784 bytes 2023-12-08T20:37:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 255980544 bytes 2023-12-08T20:37:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 282546176 bytes 2023-12-08T20:37:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 299141120 bytes 2023-12-08T20:37:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 400667648 bytes 2023-12-08T20:37:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 412555264 bytes 2023-12-08T20:37:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 492507136 bytes 2023-12-08T20:37:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 552830976 bytes 2023-12-08T20:37:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 562606080 bytes 2023-12-08T20:37:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 610604032 bytes 2023-12-08T20:37:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 636091392 bytes 2023-12-08T20:37:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 678427648 bytes 2023-12-08T20:37:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 742066176 bytes 2023-12-08T20:37:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 807752704 bytes 2023-12-08T20:37:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 888114176 bytes 2023-12-08T20:37:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 941901824 bytes 2023-12-08T20:37:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1012029440 bytes 2023-12-08T20:37:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1078162432 bytes 2023-12-08T20:37:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1109478400 bytes 2023-12-08T20:37:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1179836416 bytes 2023-12-08T20:37:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1210913792 bytes 2023-12-08T20:37:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1262568448 bytes 2023-12-08T20:37:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1290285056 bytes 2023-12-08T20:37:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1322168320 bytes 2023-12-08T20:37:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1392001024 bytes 2023-12-08T20:37:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1444767744 bytes 2023-12-08T20:37:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1501381632 bytes 2023-12-08T20:37:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1510171648 bytes 2023-12-08T20:37:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1568174080 bytes 2023-12-08T20:37:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1594512384 bytes 2023-12-08T20:37:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1617451008 bytes 2023-12-08T20:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1618060288 bytes 2023-12-08T20:37:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1731630080 bytes 2023-12-08T20:37:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1759675392 bytes 2023-12-08T20:37:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1814804480 bytes 2023-12-08T20:37:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1842742272 bytes 2023-12-08T20:37:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1905201152 bytes 2023-12-08T20:37:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_13 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2023-12-08T20:37:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:37:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-08T20:37:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:37:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-08T20:37:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:37:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_13': 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12f2c83cb34a 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12f2c83cb34a 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:12 2023 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:12 2023 2023-12-08T20:37:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:12 2023 2023-12-08T20:37:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:37:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:37:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bf4784b3-1a62-4451-b049-ff2a5cbc99eb 2023-12-08T20:37:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-08T20:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:37:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:37:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:37:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:37:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:37:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 68463616 bytes 2023-12-08T20:37:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 99605504 bytes 2023-12-08T20:37:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 146197504 bytes 2023-12-08T20:37:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 175489024 bytes 2023-12-08T20:37:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 212024320 bytes 2023-12-08T20:37:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 243507200 bytes 2023-12-08T20:37:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 346461184 bytes 2023-12-08T20:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 374582272 bytes 2023-12-08T20:37:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 418694144 bytes 2023-12-08T20:37:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 450689024 bytes 2023-12-08T20:37:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 483414016 bytes 2023-12-08T20:37:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 497388544 bytes 2023-12-08T20:37:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 515415040 bytes 2023-12-08T20:37:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 563922944 bytes 2023-12-08T20:37:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 585501696 bytes 2023-12-08T20:37:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 586150912 bytes 2023-12-08T20:37:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 586785792 bytes 2023-12-08T20:37:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 607592448 bytes 2023-12-08T20:37:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 671350784 bytes 2023-12-08T20:37:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 685625344 bytes 2023-12-08T20:37:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 712730624 bytes 2023-12-08T20:37:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 753385472 bytes 2023-12-08T20:37:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 776629248 bytes 2023-12-08T20:37:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 861485056 bytes 2023-12-08T20:37:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 909565952 bytes 2023-12-08T20:37:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 942906368 bytes 2023-12-08T20:37:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 967686144 bytes 2023-12-08T20:37:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1002616832 bytes 2023-12-08T20:37:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1029824512 bytes 2023-12-08T20:37:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1108190208 bytes 2023-12-08T20:37:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1196854272 bytes 2023-12-08T20:37:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1226898432 bytes 2023-12-08T20:37:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1266198528 bytes 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_14 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2023-12-08T20:37:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:37:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-08T20:37:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:37:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-08T20:37:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:37:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-08T20:37:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_14': 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 12fe96765c0d 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.12fe96765c0d 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:18 2023 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:18 2023 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:18 2023 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:37:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:37:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f7b0e482-81c1-456d-bbf0-de74514a24bc 2023-12-08T20:37:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:37:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:37:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-08T20:37:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:37:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:37:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:37:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:37:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:37:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 33031168 bytes 2023-12-08T20:37:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 47109120 bytes 2023-12-08T20:37:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 82677760 bytes 2023-12-08T20:37:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 97755136 bytes 2023-12-08T20:37:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 113928192 bytes 2023-12-08T20:37:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 230383616 bytes 2023-12-08T20:37:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 259434496 bytes 2023-12-08T20:37:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 295627776 bytes 2023-12-08T20:37:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 372672512 bytes 2023-12-08T20:37:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 440997888 bytes 2023-12-08T20:37:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 473911296 bytes 2023-12-08T20:37:21.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 492558336 bytes 2023-12-08T20:37:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 546059264 bytes 2023-12-08T20:37:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 560243712 bytes 2023-12-08T20:37:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 646307840 bytes 2023-12-08T20:37:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 669045760 bytes 2023-12-08T20:37:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 719796224 bytes 2023-12-08T20:37:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 785776640 bytes 2023-12-08T20:37:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 816889856 bytes 2023-12-08T20:37:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 877675520 bytes 2023-12-08T20:37:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 881122304 bytes 2023-12-08T20:37:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 895611904 bytes 2023-12-08T20:37:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 927665152 bytes 2023-12-08T20:37:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 965435392 bytes 2023-12-08T20:37:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 977707008 bytes 2023-12-08T20:37:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 994855936 bytes 2023-12-08T20:37:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 996166656 bytes 2023-12-08T20:37:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1116909568 bytes 2023-12-08T20:37:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1153810432 bytes 2023-12-08T20:37:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1181210624 bytes 2023-12-08T20:37:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1201047552 bytes 2023-12-08T20:37:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1234972672 bytes 2023-12-08T20:37:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:37:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:37:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:37:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:37:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_15 2023-12-08T20:37:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:37:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:37:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:37:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2023-12-08T20:37:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:37:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-08T20:37:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:37:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-08T20:37:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:37:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_15': 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 130add65eb23 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.130add65eb23 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:37:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:37:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:24 2023 2023-12-08T20:37:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:24 2023 2023-12-08T20:37:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:24 2023 2023-12-08T20:37:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:37:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:37:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: c614153b-e9e9-4295-a51d-042466d7263b 2023-12-08T20:37:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:37:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:37:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-08T20:37:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:37:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=13 2023-12-08T20:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:37:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 600064 bytes 2023-12-08T20:37:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 9524224 bytes 2023-12-08T20:37:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 25803776 bytes 2023-12-08T20:37:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 81477632 bytes 2023-12-08T20:37:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 100784128 bytes 2023-12-08T20:37:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 170128384 bytes 2023-12-08T20:37:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 223926272 bytes 2023-12-08T20:37:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 295176192 bytes 2023-12-08T20:37:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 380669952 bytes 2023-12-08T20:37:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 413893632 bytes 2023-12-08T20:37:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 424680448 bytes 2023-12-08T20:37:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 487989248 bytes 2023-12-08T20:37:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 508360704 bytes 2023-12-08T20:37:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 569943040 bytes 2023-12-08T20:37:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 640275456 bytes 2023-12-08T20:37:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 714840064 bytes 2023-12-08T20:37:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 737371136 bytes 2023-12-08T20:37:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 774822912 bytes 2023-12-08T20:37:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 775447552 bytes 2023-12-08T20:37:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 800631808 bytes 2023-12-08T20:37:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 860313600 bytes 2023-12-08T20:37:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 881802240 bytes 2023-12-08T20:37:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 899445760 bytes 2023-12-08T20:37:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 920416256 bytes 2023-12-08T20:37:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 974747648 bytes 2023-12-08T20:37:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 995206144 bytes 2023-12-08T20:37:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1033266176 bytes 2023-12-08T20:37:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1078192128 bytes 2023-12-08T20:37:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1142754304 bytes 2023-12-08T20:37:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1165837312 bytes 2023-12-08T20:37:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1212926976 bytes 2023-12-08T20:37:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1232863232 bytes 2023-12-08T20:37:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1233466368 bytes 2023-12-08T20:37:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1368713216 bytes 2023-12-08T20:37:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1389326336 bytes 2023-12-08T20:37:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1412711424 bytes 2023-12-08T20:37:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1489630208 bytes 2023-12-08T20:37:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1500081152 bytes 2023-12-08T20:37:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1521985536 bytes 2023-12-08T20:37:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1530352640 bytes 2023-12-08T20:37:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1611415552 bytes 2023-12-08T20:37:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1616063488 bytes 2023-12-08T20:37:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1650652160 bytes 2023-12-08T20:37:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1711347712 bytes 2023-12-08T20:37:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1752371200 bytes 2023-12-08T20:37:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1802117120 bytes 2023-12-08T20:37:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1833769984 bytes 2023-12-08T20:37:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1859499008 bytes 2023-12-08T20:37:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1893249024 bytes 2023-12-08T20:37:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 1918315520 bytes 2023-12-08T20:37:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 1935796224 bytes 2023-12-08T20:37:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 1972990976 bytes 2023-12-08T20:37:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2020287488 bytes 2023-12-08T20:37:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2023382016 bytes 2023-12-08T20:37:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2094307328 bytes 2023-12-08T20:37:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2102291456 bytes 2023-12-08T20:37:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2136430592 bytes 2023-12-08T20:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2157022208 bytes 2023-12-08T20:37:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2180068352 bytes 2023-12-08T20:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2189957120 bytes 2023-12-08T20:37:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2226717696 bytes 2023-12-08T20:37:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2318778368 bytes 2023-12-08T20:37:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2363929600 bytes 2023-12-08T20:37:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2409354240 bytes 2023-12-08T20:37:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2447997952 bytes 2023-12-08T20:37:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2498239488 bytes 2023-12-08T20:37:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2546753536 bytes 2023-12-08T20:37:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2616923136 bytes 2023-12-08T20:37:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 2633133056 bytes 2023-12-08T20:37:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 2665132032 bytes 2023-12-08T20:37:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 2697412608 bytes 2023-12-08T20:37:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 2777563136 bytes 2023-12-08T20:37:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 2851454976 bytes 2023-12-08T20:37:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 2857952256 bytes 2023-12-08T20:37:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 2946711552 bytes 2023-12-08T20:37:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 2959665152 bytes 2023-12-08T20:37:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 2991277056 bytes 2023-12-08T20:37:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3012081664 bytes 2023-12-08T20:37:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3067389952 bytes 2023-12-08T20:37:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3125677056 bytes 2023-12-08T20:37:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3163985920 bytes 2023-12-08T20:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3232456704 bytes 2023-12-08T20:37:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3274444800 bytes 2023-12-08T20:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8400 IOs, 3313597440 bytes 2023-12-08T20:37:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_16 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2023-12-08T20:37:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:37:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-08T20:37:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:37:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-08T20:37:38.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6714907061939484) > weight (0.25) 2023-12-08T20:37:38.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.838881713445455) > weight (0.25) 2023-12-08T20:37:38.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.46270751574099944) > weight (0.25) 2023-12-08T20:37:38.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9457471733676954) > weight (0.25) 2023-12-08T20:37:38.469 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-08T20:37:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:37:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-08T20:37:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_16': 2023-12-08T20:37:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:37:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:37:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:37:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 131564bbf7f2 2023-12-08T20:37:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.131564bbf7f2 2023-12-08T20:37:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:37:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:38 2023 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:38 2023 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:38 2023 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:37:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: b5dd2e11-6d62-42d1-b7e9-d63f11c0fc3c 2023-12-08T20:37:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-08T20:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:37:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 16577536 bytes 2023-12-08T20:37:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 73938944 bytes 2023-12-08T20:37:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 113414144 bytes 2023-12-08T20:37:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 160763904 bytes 2023-12-08T20:37:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 218586112 bytes 2023-12-08T20:37:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 279799808 bytes 2023-12-08T20:37:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 353845248 bytes 2023-12-08T20:37:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 390588416 bytes 2023-12-08T20:37:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 428930048 bytes 2023-12-08T20:37:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 474642432 bytes 2023-12-08T20:37:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 536114176 bytes 2023-12-08T20:37:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 603508736 bytes 2023-12-08T20:37:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 628066304 bytes 2023-12-08T20:37:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 661116928 bytes 2023-12-08T20:37:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 760790016 bytes 2023-12-08T20:37:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 770094080 bytes 2023-12-08T20:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 786354176 bytes 2023-12-08T20:37:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 857767936 bytes 2023-12-08T20:37:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 876019712 bytes 2023-12-08T20:37:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 962436096 bytes 2023-12-08T20:37:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1006864384 bytes 2023-12-08T20:37:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1026075648 bytes 2023-12-08T20:37:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1063585792 bytes 2023-12-08T20:37:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1124585472 bytes 2023-12-08T20:37:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1131195392 bytes 2023-12-08T20:37:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1203278848 bytes 2023-12-08T20:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1239233536 bytes 2023-12-08T20:37:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1287120896 bytes 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_17 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2023-12-08T20:37:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:37:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-08T20:37:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:37:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-08T20:37:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:37:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-08T20:37:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_17': 2023-12-08T20:37:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:37:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:37:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1321de1e3c1c 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1321de1e3c1c 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:44 2023 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:44 2023 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:44 2023 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: d53aaa49-211e-4a25-8cfe-c474c72c1761 2023-12-08T20:37:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:37:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:37:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-08T20:37:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:37:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:37:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=6 2023-12-08T20:37:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:37:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:37:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 14212096 bytes 2023-12-08T20:37:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 14799872 bytes 2023-12-08T20:37:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 66987008 bytes 2023-12-08T20:37:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 85044224 bytes 2023-12-08T20:37:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 85646336 bytes 2023-12-08T20:37:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 107832320 bytes 2023-12-08T20:37:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 161781760 bytes 2023-12-08T20:37:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 167019520 bytes 2023-12-08T20:37:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 182084608 bytes 2023-12-08T20:37:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 220468224 bytes 2023-12-08T20:37:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 265161728 bytes 2023-12-08T20:37:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 306320384 bytes 2023-12-08T20:37:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 369553408 bytes 2023-12-08T20:37:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 402877440 bytes 2023-12-08T20:37:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 459608064 bytes 2023-12-08T20:37:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 464959488 bytes 2023-12-08T20:37:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 482118656 bytes 2023-12-08T20:37:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 510826496 bytes 2023-12-08T20:37:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 541507584 bytes 2023-12-08T20:37:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 565700608 bytes 2023-12-08T20:37:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 574146560 bytes 2023-12-08T20:37:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 599356416 bytes 2023-12-08T20:37:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 632074240 bytes 2023-12-08T20:37:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 673604608 bytes 2023-12-08T20:37:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 701369344 bytes 2023-12-08T20:37:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 744666112 bytes 2023-12-08T20:37:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 752616448 bytes 2023-12-08T20:37:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 776039424 bytes 2023-12-08T20:37:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 782387200 bytes 2023-12-08T20:37:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 838151168 bytes 2023-12-08T20:37:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 865477632 bytes 2023-12-08T20:37:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 924358656 bytes 2023-12-08T20:37:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 950793216 bytes 2023-12-08T20:37:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1009819648 bytes 2023-12-08T20:37:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1094827008 bytes 2023-12-08T20:37:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1141518336 bytes 2023-12-08T20:37:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1153782784 bytes 2023-12-08T20:37:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1221786624 bytes 2023-12-08T20:37:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1269058560 bytes 2023-12-08T20:37:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1352817664 bytes 2023-12-08T20:37:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:37:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_18 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:37:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2023-12-08T20:37:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:37:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-08T20:37:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:37:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:37:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-08T20:37:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:37:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-08T20:37:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_18': 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 132d72175c5c 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.132d72175c5c 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:51 2023 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:51 2023 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:51 2023 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:37:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:37:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bf95c8e1-6877-4faf-91e1-cabad7898b86 2023-12-08T20:37:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:37:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:37:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-08T20:37:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:37:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:37:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:37:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:37:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=14 2023-12-08T20:37:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:37:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:37:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 37035008 bytes 2023-12-08T20:37:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 74360832 bytes 2023-12-08T20:37:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 99588096 bytes 2023-12-08T20:37:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 144216064 bytes 2023-12-08T20:37:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 192584704 bytes 2023-12-08T20:37:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 209733632 bytes 2023-12-08T20:37:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 311044096 bytes 2023-12-08T20:37:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 385630208 bytes 2023-12-08T20:37:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 470818816 bytes 2023-12-08T20:37:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 527072256 bytes 2023-12-08T20:37:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 561099776 bytes 2023-12-08T20:37:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 614858752 bytes 2023-12-08T20:37:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 632456192 bytes 2023-12-08T20:37:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 689041408 bytes 2023-12-08T20:37:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 710076416 bytes 2023-12-08T20:37:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 743629824 bytes 2023-12-08T20:37:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 744250368 bytes 2023-12-08T20:37:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 796067840 bytes 2023-12-08T20:37:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 796682240 bytes 2023-12-08T20:37:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 819559424 bytes 2023-12-08T20:37:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 889628672 bytes 2023-12-08T20:37:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 918707200 bytes 2023-12-08T20:37:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 931295232 bytes 2023-12-08T20:37:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 946822144 bytes 2023-12-08T20:37:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1026400256 bytes 2023-12-08T20:37:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1086544896 bytes 2023-12-08T20:37:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1092802560 bytes 2023-12-08T20:37:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1171751936 bytes 2023-12-08T20:37:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1172377600 bytes 2023-12-08T20:37:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1267528704 bytes 2023-12-08T20:37:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1301584896 bytes 2023-12-08T20:37:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1370669056 bytes 2023-12-08T20:37:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1392776192 bytes 2023-12-08T20:37:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1434311680 bytes 2023-12-08T20:37:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1511704576 bytes 2023-12-08T20:37:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1611676672 bytes 2023-12-08T20:37:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1663620096 bytes 2023-12-08T20:37:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1717926912 bytes 2023-12-08T20:37:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1741621248 bytes 2023-12-08T20:37:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1777516544 bytes 2023-12-08T20:37:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1866642432 bytes 2023-12-08T20:37:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1982940160 bytes 2023-12-08T20:37:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2045416448 bytes 2023-12-08T20:38:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2118081536 bytes 2023-12-08T20:38:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2128557056 bytes 2023-12-08T20:38:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2179249152 bytes 2023-12-08T20:38:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2231859200 bytes 2023-12-08T20:38:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2253191168 bytes 2023-12-08T20:38:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2357046272 bytes 2023-12-08T20:38:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2438052864 bytes 2023-12-08T20:38:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2470894592 bytes 2023-12-08T20:38:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2480943104 bytes 2023-12-08T20:38:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2515118080 bytes 2023-12-08T20:38:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2552836096 bytes 2023-12-08T20:38:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2619001856 bytes 2023-12-08T20:38:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2655781888 bytes 2023-12-08T20:38:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2668427264 bytes 2023-12-08T20:38:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2682764288 bytes 2023-12-08T20:38:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2762469376 bytes 2023-12-08T20:38:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2769890304 bytes 2023-12-08T20:38:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2799521792 bytes 2023-12-08T20:38:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2880666624 bytes 2023-12-08T20:38:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2885237760 bytes 2023-12-08T20:38:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2886552576 bytes 2023-12-08T20:38:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2937329664 bytes 2023-12-08T20:38:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2991762432 bytes 2023-12-08T20:38:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 3031691264 bytes 2023-12-08T20:38:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3097052160 bytes 2023-12-08T20:38:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3114616832 bytes 2023-12-08T20:38:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3147632640 bytes 2023-12-08T20:38:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3183585280 bytes 2023-12-08T20:38:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3196335104 bytes 2023-12-08T20:38:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3224844288 bytes 2023-12-08T20:38:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3278125056 bytes 2023-12-08T20:38:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3365200896 bytes 2023-12-08T20:38:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3445522432 bytes 2023-12-08T20:38:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3479493632 bytes 2023-12-08T20:38:05.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4131445428416789) > weight (0.25) 2023-12-08T20:38:05.355 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-08T20:38:05.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-08T20:38:05.355 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-08T20:38:05.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T20:38:05.347+0000 7f4911c3e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 43627) UID: 1000 2023-12-08T20:38:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3546724352 bytes 2023-12-08T20:38:05.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:38:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3573791744 bytes 2023-12-08T20:38:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3582859264 bytes 2023-12-08T20:38:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3626580992 bytes 2023-12-08T20:38:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3646371840 bytes 2023-12-08T20:38:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3681394688 bytes 2023-12-08T20:38:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_19 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2023-12-08T20:38:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:38:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-08T20:38:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:38:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-08T20:38:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:38:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_19': 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1339736b80e8 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1339736b80e8 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:38:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:06 2023 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:06 2023 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:06 2023 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 04e61c39-ed38-426f-95be-662b7ae37286 2023-12-08T20:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:38:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:38:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-08T20:38:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:38:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:38:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=11 2023-12-08T20:38:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:38:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:38:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 53937152 bytes 2023-12-08T20:38:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 96345088 bytes 2023-12-08T20:38:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 157424640 bytes 2023-12-08T20:38:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 224901120 bytes 2023-12-08T20:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 317455360 bytes 2023-12-08T20:38:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 377225216 bytes 2023-12-08T20:38:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 377841664 bytes 2023-12-08T20:38:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 450422784 bytes 2023-12-08T20:38:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 470147072 bytes 2023-12-08T20:38:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 538958848 bytes 2023-12-08T20:38:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 584864768 bytes 2023-12-08T20:38:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 585479168 bytes 2023-12-08T20:38:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 623611904 bytes 2023-12-08T20:38:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 654748672 bytes 2023-12-08T20:38:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 687126528 bytes 2023-12-08T20:38:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 709452800 bytes 2023-12-08T20:38:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 738244608 bytes 2023-12-08T20:38:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 777252864 bytes 2023-12-08T20:38:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 799624192 bytes 2023-12-08T20:38:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 886216704 bytes 2023-12-08T20:38:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 908433408 bytes 2023-12-08T20:38:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 932446208 bytes 2023-12-08T20:38:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1013868544 bytes 2023-12-08T20:38:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1048773632 bytes 2023-12-08T20:38:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1164474368 bytes 2023-12-08T20:38:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1234916352 bytes 2023-12-08T20:38:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1257887744 bytes 2023-12-08T20:38:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1295635456 bytes 2023-12-08T20:38:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1355509760 bytes 2023-12-08T20:38:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1396636672 bytes 2023-12-08T20:38:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1425302528 bytes 2023-12-08T20:38:12.356 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-08T20:38:12.357 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:38:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:38:12.357 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-08T20:38:12.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-08T20:38:12.357 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-08T20:38:12.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-08T20:38:12.367 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-08T20:38:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1425920000 bytes 2023-12-08T20:38:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1466426368 bytes 2023-12-08T20:38:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1555130368 bytes 2023-12-08T20:38:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1560970240 bytes 2023-12-08T20:38:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1609157632 bytes 2023-12-08T20:38:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1634583552 bytes 2023-12-08T20:38:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1651469312 bytes 2023-12-08T20:38:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1694364672 bytes 2023-12-08T20:38:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1747111936 bytes 2023-12-08T20:38:13.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1805458432 bytes 2023-12-08T20:38:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1824755712 bytes 2023-12-08T20:38:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1825376256 bytes 2023-12-08T20:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1890350080 bytes 2023-12-08T20:38:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1955784704 bytes 2023-12-08T20:38:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2012557312 bytes 2023-12-08T20:38:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2048944128 bytes 2023-12-08T20:38:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2070348800 bytes 2023-12-08T20:38:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2076136448 bytes 2023-12-08T20:38:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2106018816 bytes 2023-12-08T20:38:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2111817728 bytes 2023-12-08T20:38:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2176402432 bytes 2023-12-08T20:38:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2253039616 bytes 2023-12-08T20:38:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2287673344 bytes 2023-12-08T20:38:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2306577408 bytes 2023-12-08T20:38:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2346468352 bytes 2023-12-08T20:38:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2352226304 bytes 2023-12-08T20:38:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2477985792 bytes 2023-12-08T20:38:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2538561536 bytes 2023-12-08T20:38:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2585790464 bytes 2023-12-08T20:38:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2588170240 bytes 2023-12-08T20:38:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2625836032 bytes 2023-12-08T20:38:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2689267712 bytes 2023-12-08T20:38:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2735761408 bytes 2023-12-08T20:38:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2765598720 bytes 2023-12-08T20:38:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2812146688 bytes 2023-12-08T20:38:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2831381504 bytes 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_20 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2023-12-08T20:38:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:38:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-08T20:38:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:38:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-08T20:38:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:38:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_20': 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 134f64f3759b 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.134f64f3759b 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:38:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:18 2023 2023-12-08T20:38:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:18 2023 2023-12-08T20:38:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:18 2023 2023-12-08T20:38:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:38:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:38:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 9efe6779-be3c-472b-840a-27713ee6053d 2023-12-08T20:38:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:38:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:38:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-08T20:38:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:38:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:38:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:38:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:38:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:38:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 103080960 bytes 2023-12-08T20:38:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 162435072 bytes 2023-12-08T20:38:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 172286976 bytes 2023-12-08T20:38:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 267626496 bytes 2023-12-08T20:38:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 305596416 bytes 2023-12-08T20:38:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 363109376 bytes 2023-12-08T20:38:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 398031872 bytes 2023-12-08T20:38:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 428069888 bytes 2023-12-08T20:38:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 482919424 bytes 2023-12-08T20:38:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 493524992 bytes 2023-12-08T20:38:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 527063040 bytes 2023-12-08T20:38:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 563357696 bytes 2023-12-08T20:38:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 614359040 bytes 2023-12-08T20:38:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 648110080 bytes 2023-12-08T20:38:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 692889600 bytes 2023-12-08T20:38:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 770736128 bytes 2023-12-08T20:38:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 802733056 bytes 2023-12-08T20:38:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 844266496 bytes 2023-12-08T20:38:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 905374720 bytes 2023-12-08T20:38:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 944530432 bytes 2023-12-08T20:38:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 980518912 bytes 2023-12-08T20:38:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1006402560 bytes 2023-12-08T20:38:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1017730048 bytes 2023-12-08T20:38:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1037507584 bytes 2023-12-08T20:38:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1086632960 bytes 2023-12-08T20:38:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1181356032 bytes 2023-12-08T20:38:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1190829056 bytes 2023-12-08T20:38:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1205499904 bytes 2023-12-08T20:38:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1234235392 bytes 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_21 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:38:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:38:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2023-12-08T20:38:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:38:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-08T20:38:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:38:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-08T20:38:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:38:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-08T20:38:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_21': 2023-12-08T20:38:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:38:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:38:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 135b79e7fa44 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.135b79e7fa44 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:24 2023 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:24 2023 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:24 2023 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f344c1db-e806-48ca-a049-9d5faa56ca26 2023-12-08T20:38:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:38:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:38:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-08T20:38:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:38:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:38:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:38:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:38:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:38:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 617472 bytes 2023-12-08T20:38:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 29403136 bytes 2023-12-08T20:38:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 30280704 bytes 2023-12-08T20:38:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 77742080 bytes 2023-12-08T20:38:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 102980608 bytes 2023-12-08T20:38:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 103570432 bytes 2023-12-08T20:38:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 131725312 bytes 2023-12-08T20:38:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 160939008 bytes 2023-12-08T20:38:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 203835392 bytes 2023-12-08T20:38:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 263528448 bytes 2023-12-08T20:38:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 328981504 bytes 2023-12-08T20:38:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 384757760 bytes 2023-12-08T20:38:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 418558976 bytes 2023-12-08T20:38:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 477702144 bytes 2023-12-08T20:38:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 520057856 bytes 2023-12-08T20:38:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 564198400 bytes 2023-12-08T20:38:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 611777536 bytes 2023-12-08T20:38:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 637099008 bytes 2023-12-08T20:38:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 679340032 bytes 2023-12-08T20:38:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 690889728 bytes 2023-12-08T20:38:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 702405632 bytes 2023-12-08T20:38:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 731863040 bytes 2023-12-08T20:38:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 768021504 bytes 2023-12-08T20:38:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 786038784 bytes 2023-12-08T20:38:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 823021568 bytes 2023-12-08T20:38:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 858551296 bytes 2023-12-08T20:38:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 869190656 bytes 2023-12-08T20:38:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 908731392 bytes 2023-12-08T20:38:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 991814656 bytes 2023-12-08T20:38:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1000365056 bytes 2023-12-08T20:38:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1052055552 bytes 2023-12-08T20:38:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1090156544 bytes 2023-12-08T20:38:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1163862016 bytes 2023-12-08T20:38:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1236072448 bytes 2023-12-08T20:38:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1299506176 bytes 2023-12-08T20:38:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1339019264 bytes 2023-12-08T20:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1355609088 bytes 2023-12-08T20:38:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1373114368 bytes 2023-12-08T20:38:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1446231040 bytes 2023-12-08T20:38:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1533508608 bytes 2023-12-08T20:38:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1562798080 bytes 2023-12-08T20:38:32.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1660028928 bytes 2023-12-08T20:38:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1660644352 bytes 2023-12-08T20:38:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1697060864 bytes 2023-12-08T20:38:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1758973952 bytes 2023-12-08T20:38:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1763364864 bytes 2023-12-08T20:38:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1821426688 bytes 2023-12-08T20:38:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1887890432 bytes 2023-12-08T20:38:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1897272320 bytes 2023-12-08T20:38:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 1934603264 bytes 2023-12-08T20:38:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2038891520 bytes 2023-12-08T20:38:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2096861184 bytes 2023-12-08T20:38:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2159525888 bytes 2023-12-08T20:38:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2201914368 bytes 2023-12-08T20:38:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_22 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:38:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2023-12-08T20:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-08T20:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-08T20:38:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:38:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-08T20:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_22': 2023-12-08T20:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1367b72809fd 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1367b72809fd 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:34 2023 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:34 2023 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:34 2023 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:38:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 23fc213f-3b3f-4a6e-8700-6338e36a234a 2023-12-08T20:38:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-08T20:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:38:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=14 2023-12-08T20:38:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:38:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:38:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 58136576 bytes 2023-12-08T20:38:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 100893696 bytes 2023-12-08T20:38:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 101484544 bytes 2023-12-08T20:38:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 139211776 bytes 2023-12-08T20:38:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 173691904 bytes 2023-12-08T20:38:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 255142912 bytes 2023-12-08T20:38:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 281849856 bytes 2023-12-08T20:38:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 314292224 bytes 2023-12-08T20:38:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 371072000 bytes 2023-12-08T20:38:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 428146688 bytes 2023-12-08T20:38:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 473985024 bytes 2023-12-08T20:38:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 518700032 bytes 2023-12-08T20:38:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 551749632 bytes 2023-12-08T20:38:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 577101824 bytes 2023-12-08T20:38:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 613187584 bytes 2023-12-08T20:38:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 682817536 bytes 2023-12-08T20:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 720858112 bytes 2023-12-08T20:38:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 798284800 bytes 2023-12-08T20:38:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 853263360 bytes 2023-12-08T20:38:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 909794304 bytes 2023-12-08T20:38:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 978441216 bytes 2023-12-08T20:38:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 979833856 bytes 2023-12-08T20:38:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1007027200 bytes 2023-12-08T20:38:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1013456896 bytes 2023-12-08T20:38:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1058670592 bytes 2023-12-08T20:38:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1077239808 bytes 2023-12-08T20:38:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1133285376 bytes 2023-12-08T20:38:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1218234368 bytes 2023-12-08T20:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1292681216 bytes 2023-12-08T20:38:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1340354560 bytes 2023-12-08T20:38:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1350395904 bytes 2023-12-08T20:38:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1409864704 bytes 2023-12-08T20:38:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1468805120 bytes 2023-12-08T20:38:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1495720960 bytes 2023-12-08T20:38:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1559002112 bytes 2023-12-08T20:38:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1592657920 bytes 2023-12-08T20:38:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1634279424 bytes 2023-12-08T20:38:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1673996288 bytes 2023-12-08T20:38:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1694247936 bytes 2023-12-08T20:38:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1698122752 bytes 2023-12-08T20:38:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1761715200 bytes 2023-12-08T20:38:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1859140608 bytes 2023-12-08T20:38:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1876738048 bytes 2023-12-08T20:38:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1917446144 bytes 2023-12-08T20:38:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1948507136 bytes 2023-12-08T20:38:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1990455296 bytes 2023-12-08T20:38:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2084434944 bytes 2023-12-08T20:38:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2162194432 bytes 2023-12-08T20:38:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2170385408 bytes 2023-12-08T20:38:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2190254080 bytes 2023-12-08T20:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2291810304 bytes 2023-12-08T20:38:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2374035456 bytes 2023-12-08T20:38:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2424918016 bytes 2023-12-08T20:38:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2484674560 bytes 2023-12-08T20:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2529325056 bytes 2023-12-08T20:38:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2562527232 bytes 2023-12-08T20:38:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2597572608 bytes 2023-12-08T20:38:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2644090880 bytes 2023-12-08T20:38:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2723988480 bytes 2023-12-08T20:38:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2731789312 bytes 2023-12-08T20:38:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2786847744 bytes 2023-12-08T20:38:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2855131136 bytes 2023-12-08T20:38:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2855752704 bytes 2023-12-08T20:38:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2942016512 bytes 2023-12-08T20:38:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2982544384 bytes 2023-12-08T20:38:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 3040344064 bytes 2023-12-08T20:38:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 3075908608 bytes 2023-12-08T20:38:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3101821952 bytes 2023-12-08T20:38:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3205955584 bytes 2023-12-08T20:38:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3229582336 bytes 2023-12-08T20:38:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3236340736 bytes 2023-12-08T20:38:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3296390144 bytes 2023-12-08T20:38:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3339200512 bytes 2023-12-08T20:38:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3343474688 bytes 2023-12-08T20:38:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3431177216 bytes 2023-12-08T20:38:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3501723648 bytes 2023-12-08T20:38:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3530745856 bytes 2023-12-08T20:38:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3541988352 bytes 2023-12-08T20:38:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_23 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:38:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2023-12-08T20:38:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:38:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-08T20:38:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:38:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-08T20:38:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:38:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-08T20:38:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_23': 2023-12-08T20:38:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:38:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:38:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:38:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13724ac8e35a 2023-12-08T20:38:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13724ac8e35a 2023-12-08T20:38:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:49 2023 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:49 2023 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:49 2023 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: d02d2b15-17e5-49e1-a2bb-193d056e8988 2023-12-08T20:38:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:38:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:38:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-08T20:38:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:38:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:38:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=7 2023-12-08T20:38:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:38:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:38:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 91366400 bytes 2023-12-08T20:38:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 142821376 bytes 2023-12-08T20:38:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 188859392 bytes 2023-12-08T20:38:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 224468992 bytes 2023-12-08T20:38:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 253910016 bytes 2023-12-08T20:38:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 293123072 bytes 2023-12-08T20:38:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 355202048 bytes 2023-12-08T20:38:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 438305792 bytes 2023-12-08T20:38:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 438935552 bytes 2023-12-08T20:38:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 474842112 bytes 2023-12-08T20:38:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 499593216 bytes 2023-12-08T20:38:52.470 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5991625201455434) > weight (0.25) 2023-12-08T20:38:52.470 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-08T20:38:52.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-08T20:38:52.471 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-08T20:38:52.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:38:52.462+0000 7f2a19904700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 41404) UID: 1000 2023-12-08T20:38:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 603601920 bytes 2023-12-08T20:38:52.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:38:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 608448512 bytes 2023-12-08T20:38:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 636726272 bytes 2023-12-08T20:38:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 701535232 bytes 2023-12-08T20:38:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 766949376 bytes 2023-12-08T20:38:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 767576064 bytes 2023-12-08T20:38:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 794266624 bytes 2023-12-08T20:38:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 837573632 bytes 2023-12-08T20:38:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 838185984 bytes 2023-12-08T20:38:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 876470272 bytes 2023-12-08T20:38:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 883097600 bytes 2023-12-08T20:38:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 935102464 bytes 2023-12-08T20:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1044500480 bytes 2023-12-08T20:38:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1123923968 bytes 2023-12-08T20:38:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1196755968 bytes 2023-12-08T20:38:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1272131584 bytes 2023-12-08T20:38:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1317966848 bytes 2023-12-08T20:38:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1365563392 bytes 2023-12-08T20:38:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1437079552 bytes 2023-12-08T20:38:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1473410048 bytes 2023-12-08T20:38:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1536780288 bytes 2023-12-08T20:38:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1537393664 bytes 2023-12-08T20:38:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1538023424 bytes 2023-12-08T20:38:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1609700352 bytes 2023-12-08T20:38:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1610386432 bytes 2023-12-08T20:38:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1649401856 bytes 2023-12-08T20:38:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1704123392 bytes 2023-12-08T20:38:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_24 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:38:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2023-12-08T20:38:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:38:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-08T20:38:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:38:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:38:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-08T20:38:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:38:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-08T20:38:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_24': 2023-12-08T20:38:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:38:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:38:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:38:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 137eb162084d 2023-12-08T20:38:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.137eb162084d 2023-12-08T20:38:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:38:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:38:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:38:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:38:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:57 2023 2023-12-08T20:38:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:57 2023 2023-12-08T20:38:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:57 2023 2023-12-08T20:38:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:38:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:38:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0417c7e3-5aea-47f6-9a2e-317d60062a37 2023-12-08T20:38:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:38:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:38:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-08T20:38:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:38:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:38:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:38:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:38:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:38:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:38:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:38:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 44726272 bytes 2023-12-08T20:38:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 60105728 bytes 2023-12-08T20:38:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 133343232 bytes 2023-12-08T20:38:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 175727616 bytes 2023-12-08T20:38:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 281974784 bytes 2023-12-08T20:38:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 288316416 bytes 2023-12-08T20:38:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 334114816 bytes 2023-12-08T20:38:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 334722048 bytes 2023-12-08T20:38:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 351968256 bytes 2023-12-08T20:39:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 421372928 bytes 2023-12-08T20:39:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 484695040 bytes 2023-12-08T20:39:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 513057792 bytes 2023-12-08T20:39:00.471 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-08T20:39:00.471 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:39:00.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:39:00.472 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-08T20:39:00.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-08T20:39:00.472 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-08T20:39:00.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-08T20:39:00.475 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-12-08T20:39:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 542278656 bytes 2023-12-08T20:39:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 583134208 bytes 2023-12-08T20:39:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 630774784 bytes 2023-12-08T20:39:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 725463040 bytes 2023-12-08T20:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 757910528 bytes 2023-12-08T20:39:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 780624896 bytes 2023-12-08T20:39:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 835017728 bytes 2023-12-08T20:39:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 843023360 bytes 2023-12-08T20:39:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 862031872 bytes 2023-12-08T20:39:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 889918464 bytes 2023-12-08T20:39:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 966721536 bytes 2023-12-08T20:39:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1013466112 bytes 2023-12-08T20:39:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1081022464 bytes 2023-12-08T20:39:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1144801280 bytes 2023-12-08T20:39:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1204658176 bytes 2023-12-08T20:39:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1264692224 bytes 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_25 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:39:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2023-12-08T20:39:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-08T20:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-08T20:39:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:39:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-08T20:39:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_25': 2023-12-08T20:39:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 138e4bd1d07 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.138e4bd1d07 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:03 2023 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:03 2023 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:03 2023 2023-12-08T20:39:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:39:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:39:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0cc3114a-15b3-4921-9172-57219de8ca5a 2023-12-08T20:39:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:39:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:39:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-08T20:39:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:39:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:39:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=10 2023-12-08T20:39:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:39:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:39:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 67527680 bytes 2023-12-08T20:39:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 103852032 bytes 2023-12-08T20:39:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 148048896 bytes 2023-12-08T20:39:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 208403456 bytes 2023-12-08T20:39:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 245189632 bytes 2023-12-08T20:39:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 270302208 bytes 2023-12-08T20:39:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 329982976 bytes 2023-12-08T20:39:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 370946048 bytes 2023-12-08T20:39:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 407115776 bytes 2023-12-08T20:39:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 467843072 bytes 2023-12-08T20:39:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 495918080 bytes 2023-12-08T20:39:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 512769024 bytes 2023-12-08T20:39:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 555058176 bytes 2023-12-08T20:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 590364672 bytes 2023-12-08T20:39:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 642709504 bytes 2023-12-08T20:39:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 663457792 bytes 2023-12-08T20:39:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 746264576 bytes 2023-12-08T20:39:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 764939264 bytes 2023-12-08T20:39:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 770777088 bytes 2023-12-08T20:39:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 812859392 bytes 2023-12-08T20:39:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 839378944 bytes 2023-12-08T20:39:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 885818368 bytes 2023-12-08T20:39:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 887034880 bytes 2023-12-08T20:39:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 908525568 bytes 2023-12-08T20:39:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 970292224 bytes 2023-12-08T20:39:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 991344640 bytes 2023-12-08T20:39:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 992083968 bytes 2023-12-08T20:39:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1001942016 bytes 2023-12-08T20:39:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1021755392 bytes 2023-12-08T20:39:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1080334336 bytes 2023-12-08T20:39:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1122942976 bytes 2023-12-08T20:39:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1141373952 bytes 2023-12-08T20:39:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1186485248 bytes 2023-12-08T20:39:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1242824704 bytes 2023-12-08T20:39:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1319169024 bytes 2023-12-08T20:39:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1364670464 bytes 2023-12-08T20:39:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1398609920 bytes 2023-12-08T20:39:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1515275264 bytes 2023-12-08T20:39:10.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1533601792 bytes 2023-12-08T20:39:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1618530304 bytes 2023-12-08T20:39:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1649418240 bytes 2023-12-08T20:39:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1702753280 bytes 2023-12-08T20:39:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1782028288 bytes 2023-12-08T20:39:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1847928832 bytes 2023-12-08T20:39:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1857938432 bytes 2023-12-08T20:39:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1896906752 bytes 2023-12-08T20:39:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1900066816 bytes 2023-12-08T20:39:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1936166912 bytes 2023-12-08T20:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1979521024 bytes 2023-12-08T20:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2059419648 bytes 2023-12-08T20:39:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2099542016 bytes 2023-12-08T20:39:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2158459904 bytes 2023-12-08T20:39:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2196493312 bytes 2023-12-08T20:39:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2292113408 bytes 2023-12-08T20:39:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2326393856 bytes 2023-12-08T20:39:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2347510784 bytes 2023-12-08T20:39:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2358909952 bytes 2023-12-08T20:39:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2387821568 bytes 2023-12-08T20:39:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2463666176 bytes 2023-12-08T20:39:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2498911232 bytes 2023-12-08T20:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:39:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_26 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2023-12-08T20:39:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:39:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-08T20:39:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:39:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-08T20:39:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:39:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-08T20:39:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_26': 2023-12-08T20:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13992972fdc9 2023-12-08T20:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13992972fdc9 2023-12-08T20:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:39:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:39:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:39:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:39:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:14 2023 2023-12-08T20:39:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:14 2023 2023-12-08T20:39:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:14 2023 2023-12-08T20:39:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:39:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:39:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 83bb9742-972b-4f04-9ef2-ea6466b8f121 2023-12-08T20:39:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:39:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:39:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-08T20:39:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:39:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:39:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:39:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:39:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:39:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 45808640 bytes 2023-12-08T20:39:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 47037440 bytes 2023-12-08T20:39:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 51017728 bytes 2023-12-08T20:39:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 79657984 bytes 2023-12-08T20:39:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 120505344 bytes 2023-12-08T20:39:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 139788288 bytes 2023-12-08T20:39:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 233709568 bytes 2023-12-08T20:39:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 311478272 bytes 2023-12-08T20:39:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 369901568 bytes 2023-12-08T20:39:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 414939136 bytes 2023-12-08T20:39:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 415576064 bytes 2023-12-08T20:39:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 426619904 bytes 2023-12-08T20:39:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 501254144 bytes 2023-12-08T20:39:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 521184256 bytes 2023-12-08T20:39:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 561868800 bytes 2023-12-08T20:39:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 603993088 bytes 2023-12-08T20:39:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 657395712 bytes 2023-12-08T20:39:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 724492288 bytes 2023-12-08T20:39:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 774719488 bytes 2023-12-08T20:39:18.461 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:39:18.454+0000 7fb534865700 -1 reset not still connected to 0x558a202abc70 2023-12-08T20:39:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 775339008 bytes 2023-12-08T20:39:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 816160768 bytes 2023-12-08T20:39:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 887259136 bytes 2023-12-08T20:39:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 938055680 bytes 2023-12-08T20:39:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 951702528 bytes 2023-12-08T20:39:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1007271936 bytes 2023-12-08T20:39:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1017453568 bytes 2023-12-08T20:39:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1100939264 bytes 2023-12-08T20:39:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1105886208 bytes 2023-12-08T20:39:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1117879296 bytes 2023-12-08T20:39:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1175120896 bytes 2023-12-08T20:39:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1210817536 bytes 2023-12-08T20:39:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1254990848 bytes 2023-12-08T20:39:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1287272448 bytes 2023-12-08T20:39:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1320400896 bytes 2023-12-08T20:39:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1360623616 bytes 2023-12-08T20:39:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1446930432 bytes 2023-12-08T20:39:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1481111552 bytes 2023-12-08T20:39:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1537183744 bytes 2023-12-08T20:39:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1589099520 bytes 2023-12-08T20:39:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1643238400 bytes 2023-12-08T20:39:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1666205696 bytes 2023-12-08T20:39:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1690924032 bytes 2023-12-08T20:39:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1717139456 bytes 2023-12-08T20:39:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1749139456 bytes 2023-12-08T20:39:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1780278272 bytes 2023-12-08T20:39:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1845830656 bytes 2023-12-08T20:39:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1886096384 bytes 2023-12-08T20:39:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1914935296 bytes 2023-12-08T20:39:23.368 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7232644184232565) > weight (0.25) 2023-12-08T20:39:23.369 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-08T20:39:23.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-08T20:39:23.369 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-08T20:39:23.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T20:39:23.362+0000 7f15ed64f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 44614) UID: 1000 2023-12-08T20:39:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1969510400 bytes 2023-12-08T20:39:23.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:39:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2067127296 bytes 2023-12-08T20:39:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2118945792 bytes 2023-12-08T20:39:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2232388608 bytes 2023-12-08T20:39:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:39:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_27 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-08T20:39:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2023-12-08T20:39:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:39:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-08T20:39:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:39:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-08T20:39:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:39:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_27': 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13a411e398db 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13a411e398db 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:39:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:39:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:24 2023 2023-12-08T20:39:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:24 2023 2023-12-08T20:39:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:24 2023 2023-12-08T20:39:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:39:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:39:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 47bc65ba-c977-4038-853c-dcaeac323616 2023-12-08T20:39:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:39:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:39:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-08T20:39:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:39:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:39:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=6 2023-12-08T20:39:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:39:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:39:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 79757312 bytes 2023-12-08T20:39:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 80391168 bytes 2023-12-08T20:39:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 145697792 bytes 2023-12-08T20:39:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 176267264 bytes 2023-12-08T20:39:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 213493760 bytes 2023-12-08T20:39:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 214118400 bytes 2023-12-08T20:39:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 241755136 bytes 2023-12-08T20:39:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 267157504 bytes 2023-12-08T20:39:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 298159104 bytes 2023-12-08T20:39:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 316280832 bytes 2023-12-08T20:39:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 357265408 bytes 2023-12-08T20:39:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 428320768 bytes 2023-12-08T20:39:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 486939648 bytes 2023-12-08T20:39:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 548653056 bytes 2023-12-08T20:39:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 602753024 bytes 2023-12-08T20:39:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 621627392 bytes 2023-12-08T20:39:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 678680576 bytes 2023-12-08T20:39:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 715659264 bytes 2023-12-08T20:39:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 716262400 bytes 2023-12-08T20:39:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 769438720 bytes 2023-12-08T20:39:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 770056192 bytes 2023-12-08T20:39:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 834610176 bytes 2023-12-08T20:39:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 870138880 bytes 2023-12-08T20:39:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 895136768 bytes 2023-12-08T20:39:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 895747072 bytes 2023-12-08T20:39:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 975513600 bytes 2023-12-08T20:39:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 976125952 bytes 2023-12-08T20:39:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1025699840 bytes 2023-12-08T20:39:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1115971584 bytes 2023-12-08T20:39:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1157595136 bytes 2023-12-08T20:39:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1180338176 bytes 2023-12-08T20:39:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1228918784 bytes 2023-12-08T20:39:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1304829952 bytes 2023-12-08T20:39:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1375992832 bytes 2023-12-08T20:39:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1399124992 bytes 2023-12-08T20:39:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1427075072 bytes 2023-12-08T20:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:39:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_28 2023-12-08T20:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-08T20:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:39:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:39:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-08T20:39:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:39:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:39:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:39:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2023-12-08T20:39:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:39:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-08T20:39:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:39:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-08T20:39:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:39:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-08T20:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_28': 2023-12-08T20:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:39:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13aefe5afeb5 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13aefe5afeb5 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:31 2023 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:31 2023 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:31 2023 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: da955d28-5776-48a9-8c45-194a8e053c29 2023-12-08T20:39:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:39:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:39:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-08T20:39:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:39:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:39:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=8 2023-12-08T20:39:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:39:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:39:32.370 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-08T20:39:32.370 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:39:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:39:32.371 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-08T20:39:32.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-08T20:39:32.371 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-08T20:39:32.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-08T20:39:32.373 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-08T20:39:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 602112 bytes 2023-12-08T20:39:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 30577664 bytes 2023-12-08T20:39:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 61773824 bytes 2023-12-08T20:39:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 166008832 bytes 2023-12-08T20:39:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 185469952 bytes 2023-12-08T20:39:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 239370240 bytes 2023-12-08T20:39:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 276566016 bytes 2023-12-08T20:39:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 315677696 bytes 2023-12-08T20:39:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 339001344 bytes 2023-12-08T20:39:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 389319680 bytes 2023-12-08T20:39:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 470268928 bytes 2023-12-08T20:39:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 508023808 bytes 2023-12-08T20:39:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 536686592 bytes 2023-12-08T20:39:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 570134528 bytes 2023-12-08T20:39:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 618004480 bytes 2023-12-08T20:39:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 678638592 bytes 2023-12-08T20:39:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 697860096 bytes 2023-12-08T20:39:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 761654272 bytes 2023-12-08T20:39:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 774814720 bytes 2023-12-08T20:39:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 848174080 bytes 2023-12-08T20:39:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 894742528 bytes 2023-12-08T20:39:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 921754624 bytes 2023-12-08T20:39:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 924326912 bytes 2023-12-08T20:39:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 969869312 bytes 2023-12-08T20:39:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1029481472 bytes 2023-12-08T20:39:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1058560000 bytes 2023-12-08T20:39:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1083755520 bytes 2023-12-08T20:39:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1084385280 bytes 2023-12-08T20:39:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1085010944 bytes 2023-12-08T20:39:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1101153280 bytes 2023-12-08T20:39:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1156296704 bytes 2023-12-08T20:39:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1156921344 bytes 2023-12-08T20:39:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1184778240 bytes 2023-12-08T20:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1228225536 bytes 2023-12-08T20:39:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1242479616 bytes 2023-12-08T20:39:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1243097088 bytes 2023-12-08T20:39:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1262282752 bytes 2023-12-08T20:39:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1300487168 bytes 2023-12-08T20:39:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1322928128 bytes 2023-12-08T20:39:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1385303040 bytes 2023-12-08T20:39:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1436185600 bytes 2023-12-08T20:39:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1514426368 bytes 2023-12-08T20:39:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1562744832 bytes 2023-12-08T20:39:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1601917952 bytes 2023-12-08T20:39:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1627676672 bytes 2023-12-08T20:39:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1654148096 bytes 2023-12-08T20:39:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1723787264 bytes 2023-12-08T20:39:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1782203392 bytes 2023-12-08T20:39:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_29 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2023-12-08T20:39:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:39:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-08T20:39:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:39:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-08T20:39:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:39:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-08T20:39:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_29': 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13c2cfec8f64 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13c2cfec8f64 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:40 2023 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:40 2023 2023-12-08T20:39:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:40 2023 2023-12-08T20:39:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:39:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:39:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 9f548ada-9bca-4758-8a9d-577afa105e40 2023-12-08T20:39:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:39:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:39:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-08T20:39:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:39:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:39:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:39:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:39:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:39:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 11102208 bytes 2023-12-08T20:39:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 43533312 bytes 2023-12-08T20:39:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 67137536 bytes 2023-12-08T20:39:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 130534400 bytes 2023-12-08T20:39:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 231374848 bytes 2023-12-08T20:39:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 280375296 bytes 2023-12-08T20:39:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 391459840 bytes 2023-12-08T20:39:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 439696384 bytes 2023-12-08T20:39:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 492939264 bytes 2023-12-08T20:39:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 524111872 bytes 2023-12-08T20:39:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 582209536 bytes 2023-12-08T20:39:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 643487744 bytes 2023-12-08T20:39:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 658957312 bytes 2023-12-08T20:39:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 673653760 bytes 2023-12-08T20:39:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 721063936 bytes 2023-12-08T20:39:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 771792896 bytes 2023-12-08T20:39:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 784562176 bytes 2023-12-08T20:39:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 859886592 bytes 2023-12-08T20:39:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 894252032 bytes 2023-12-08T20:39:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 919272448 bytes 2023-12-08T20:39:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 920156160 bytes 2023-12-08T20:39:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 957989888 bytes 2023-12-08T20:39:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 992023552 bytes 2023-12-08T20:39:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1004599296 bytes 2023-12-08T20:39:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1013738496 bytes 2023-12-08T20:39:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1030810624 bytes 2023-12-08T20:39:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1102654464 bytes 2023-12-08T20:39:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1171680256 bytes 2023-12-08T20:39:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1210244096 bytes 2023-12-08T20:39:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1222446080 bytes 2023-12-08T20:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1229846528 bytes 2023-12-08T20:39:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1259255808 bytes 2023-12-08T20:39:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1282956288 bytes 2023-12-08T20:39:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1307539456 bytes 2023-12-08T20:39:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1311255552 bytes 2023-12-08T20:39:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1334181888 bytes 2023-12-08T20:39:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1365219328 bytes 2023-12-08T20:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1429240832 bytes 2023-12-08T20:39:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1512078336 bytes 2023-12-08T20:39:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1562876928 bytes 2023-12-08T20:39:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1604009984 bytes 2023-12-08T20:39:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1634853888 bytes 2023-12-08T20:39:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1663673344 bytes 2023-12-08T20:39:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1711655936 bytes 2023-12-08T20:39:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1719462912 bytes 2023-12-08T20:39:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1800353792 bytes 2023-12-08T20:39:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1821850624 bytes 2023-12-08T20:39:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1896613888 bytes 2023-12-08T20:39:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1906530304 bytes 2023-12-08T20:39:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 1952637952 bytes 2023-12-08T20:39:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2035414016 bytes 2023-12-08T20:39:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:39:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:39:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:39:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_30 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2023-12-08T20:39:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:39:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-08T20:39:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:39:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-08T20:39:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:39:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-08T20:39:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_30': 2023-12-08T20:39:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:39:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:39:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:39:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13cd86e0ddbf 2023-12-08T20:39:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13cd86e0ddbf 2023-12-08T20:39:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:39:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:39:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:50 2023 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:50 2023 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:50 2023 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 2d94a5cb-6bdc-46af-9117-bdba7256b886 2023-12-08T20:39:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-08T20:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:39:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=7 2023-12-08T20:39:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:39:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:39:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 46501888 bytes 2023-12-08T20:39:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 60641280 bytes 2023-12-08T20:39:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 111992832 bytes 2023-12-08T20:39:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 134509568 bytes 2023-12-08T20:39:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 155179008 bytes 2023-12-08T20:39:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 195512320 bytes 2023-12-08T20:39:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 246285312 bytes 2023-12-08T20:39:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 290118656 bytes 2023-12-08T20:39:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 336129024 bytes 2023-12-08T20:39:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 365979648 bytes 2023-12-08T20:39:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 414599168 bytes 2023-12-08T20:39:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 476552192 bytes 2023-12-08T20:39:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 535390208 bytes 2023-12-08T20:39:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 555475968 bytes 2023-12-08T20:39:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 600734720 bytes 2023-12-08T20:39:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 654639104 bytes 2023-12-08T20:39:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 712688640 bytes 2023-12-08T20:39:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 745791488 bytes 2023-12-08T20:39:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 778695680 bytes 2023-12-08T20:39:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 779296768 bytes 2023-12-08T20:39:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 819313664 bytes 2023-12-08T20:39:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 846391296 bytes 2023-12-08T20:39:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 889598976 bytes 2023-12-08T20:39:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 892263424 bytes 2023-12-08T20:39:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 921517056 bytes 2023-12-08T20:39:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 929843200 bytes 2023-12-08T20:39:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 995852288 bytes 2023-12-08T20:39:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1036541952 bytes 2023-12-08T20:39:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1066345472 bytes 2023-12-08T20:39:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1114503168 bytes 2023-12-08T20:39:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1170038784 bytes 2023-12-08T20:39:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1249721344 bytes 2023-12-08T20:39:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1274436608 bytes 2023-12-08T20:39:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1327562752 bytes 2023-12-08T20:39:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1385476096 bytes 2023-12-08T20:39:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1416909824 bytes 2023-12-08T20:39:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1438018560 bytes 2023-12-08T20:39:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1504278528 bytes 2023-12-08T20:39:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1510016000 bytes 2023-12-08T20:39:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1609745408 bytes 2023-12-08T20:39:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1612596224 bytes 2023-12-08T20:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:39:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_31 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-08T20:39:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2023-12-08T20:39:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:39:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-08T20:39:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:39:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:39:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-08T20:39:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:39:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-08T20:39:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_31': 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13d7ccd1fb0d 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13d7ccd1fb0d 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:58 2023 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:58 2023 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:58 2023 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:39:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 4250ab73-a265-4e1d-b62b-01a21da69296 2023-12-08T20:39:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:39:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:39:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-08T20:39:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:39:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:39:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:39:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:39:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=10 2023-12-08T20:39:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:39:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:39:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 16239616 bytes 2023-12-08T20:39:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 105660416 bytes 2023-12-08T20:39:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 153549824 bytes 2023-12-08T20:39:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 192371712 bytes 2023-12-08T20:40:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 259630080 bytes 2023-12-08T20:40:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 356009984 bytes 2023-12-08T20:40:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 428371968 bytes 2023-12-08T20:40:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 482053120 bytes 2023-12-08T20:40:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 506866688 bytes 2023-12-08T20:40:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 577475584 bytes 2023-12-08T20:40:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 618036224 bytes 2023-12-08T20:40:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 666074112 bytes 2023-12-08T20:40:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 735855616 bytes 2023-12-08T20:40:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 758149120 bytes 2023-12-08T20:40:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 850507776 bytes 2023-12-08T20:40:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 925939712 bytes 2023-12-08T20:40:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 969733120 bytes 2023-12-08T20:40:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 989529088 bytes 2023-12-08T20:40:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 990735360 bytes 2023-12-08T20:40:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1026884608 bytes 2023-12-08T20:40:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1045153792 bytes 2023-12-08T20:40:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1067503616 bytes 2023-12-08T20:40:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1121120256 bytes 2023-12-08T20:40:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1141239808 bytes 2023-12-08T20:40:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1239530496 bytes 2023-12-08T20:40:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1298650112 bytes 2023-12-08T20:40:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1327176704 bytes 2023-12-08T20:40:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1387375616 bytes 2023-12-08T20:40:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1405468672 bytes 2023-12-08T20:40:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1411561472 bytes 2023-12-08T20:40:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1465639936 bytes 2023-12-08T20:40:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1485667328 bytes 2023-12-08T20:40:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1563378688 bytes 2023-12-08T20:40:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1616923648 bytes 2023-12-08T20:40:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1663995904 bytes 2023-12-08T20:40:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1750842368 bytes 2023-12-08T20:40:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1820497920 bytes 2023-12-08T20:40:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1843813376 bytes 2023-12-08T20:40:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1876352000 bytes 2023-12-08T20:40:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1895965696 bytes 2023-12-08T20:40:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1926422528 bytes 2023-12-08T20:40:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 2017684480 bytes 2023-12-08T20:40:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2109890560 bytes 2023-12-08T20:40:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2161446912 bytes 2023-12-08T20:40:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2257688576 bytes 2023-12-08T20:40:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2290313216 bytes 2023-12-08T20:40:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2326409216 bytes 2023-12-08T20:40:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2329819136 bytes 2023-12-08T20:40:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2355951616 bytes 2023-12-08T20:40:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2406624256 bytes 2023-12-08T20:40:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2543033344 bytes 2023-12-08T20:40:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2620580864 bytes 2023-12-08T20:40:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:40:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:40:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:40:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:40:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_32 2023-12-08T20:40:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2023-12-08T20:40:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:40:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-08T20:40:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:40:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-08T20:40:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:40:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-08T20:40:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_32': 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13e1f6f3f670 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13e1f6f3f670 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:09 2023 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:09 2023 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:09 2023 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:40:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f540800b-9391-4c36-bc38-ff53eed6a35c 2023-12-08T20:40:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:40:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:40:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-08T20:40:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:40:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:40:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=11 2023-12-08T20:40:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:40:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:40:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 8294400 bytes 2023-12-08T20:40:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 12476416 bytes 2023-12-08T20:40:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 82409472 bytes 2023-12-08T20:40:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 105979904 bytes 2023-12-08T20:40:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 158210048 bytes 2023-12-08T20:40:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 191861760 bytes 2023-12-08T20:40:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 222620672 bytes 2023-12-08T20:40:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 342243328 bytes 2023-12-08T20:40:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 414806016 bytes 2023-12-08T20:40:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 452139008 bytes 2023-12-08T20:40:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 551871488 bytes 2023-12-08T20:40:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 589561856 bytes 2023-12-08T20:40:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 627957760 bytes 2023-12-08T20:40:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 656932864 bytes 2023-12-08T20:40:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 693305344 bytes 2023-12-08T20:40:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 730008576 bytes 2023-12-08T20:40:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 783456256 bytes 2023-12-08T20:40:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 884246528 bytes 2023-12-08T20:40:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 905635840 bytes 2023-12-08T20:40:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 958083072 bytes 2023-12-08T20:40:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 996755456 bytes 2023-12-08T20:40:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1024254976 bytes 2023-12-08T20:40:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1048321024 bytes 2023-12-08T20:40:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1111841792 bytes 2023-12-08T20:40:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1226294272 bytes 2023-12-08T20:40:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1230168064 bytes 2023-12-08T20:40:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1283390464 bytes 2023-12-08T20:40:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1302282240 bytes 2023-12-08T20:40:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1315406848 bytes 2023-12-08T20:40:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1348792320 bytes 2023-12-08T20:40:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1375775744 bytes 2023-12-08T20:40:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1376408576 bytes 2023-12-08T20:40:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1422842880 bytes 2023-12-08T20:40:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1484181504 bytes 2023-12-08T20:40:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1488196608 bytes 2023-12-08T20:40:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1572337664 bytes 2023-12-08T20:40:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1628769280 bytes 2023-12-08T20:40:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1716075520 bytes 2023-12-08T20:40:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1795980288 bytes 2023-12-08T20:40:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1877091328 bytes 2023-12-08T20:40:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1916587008 bytes 2023-12-08T20:40:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1973087232 bytes 2023-12-08T20:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2042743808 bytes 2023-12-08T20:40:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2074461184 bytes 2023-12-08T20:40:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2109513728 bytes 2023-12-08T20:40:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2208851968 bytes 2023-12-08T20:40:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2276041728 bytes 2023-12-08T20:40:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2288595968 bytes 2023-12-08T20:40:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2320423936 bytes 2023-12-08T20:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2369306624 bytes 2023-12-08T20:40:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2407237632 bytes 2023-12-08T20:40:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2457465856 bytes 2023-12-08T20:40:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2475304960 bytes 2023-12-08T20:40:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2504200192 bytes 2023-12-08T20:40:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2534486016 bytes 2023-12-08T20:40:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2563089408 bytes 2023-12-08T20:40:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2571055104 bytes 2023-12-08T20:40:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2595866624 bytes 2023-12-08T20:40:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2606621696 bytes 2023-12-08T20:40:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2639502336 bytes 2023-12-08T20:40:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2695014400 bytes 2023-12-08T20:40:21.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2751944704 bytes 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_33 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:40:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:40:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:40:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:40:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2023-12-08T20:40:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:40:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-08T20:40:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:40:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-08T20:40:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:40:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_33': 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13ea13eb9481 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13ea13eb9481 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:21 2023 2023-12-08T20:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:21 2023 2023-12-08T20:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:21 2023 2023-12-08T20:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bbf990e6-a3c5-4594-aa42-9c737edf1098 2023-12-08T20:40:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:40:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:40:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-08T20:40:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:40:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:40:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:40:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:40:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:40:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 36826112 bytes 2023-12-08T20:40:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 77864960 bytes 2023-12-08T20:40:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 83661824 bytes 2023-12-08T20:40:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 99688448 bytes 2023-12-08T20:40:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 150061056 bytes 2023-12-08T20:40:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 207923200 bytes 2023-12-08T20:40:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 256317440 bytes 2023-12-08T20:40:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 265254912 bytes 2023-12-08T20:40:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 336581632 bytes 2023-12-08T20:40:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 400391168 bytes 2023-12-08T20:40:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 476577792 bytes 2023-12-08T20:40:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 507645952 bytes 2023-12-08T20:40:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 520387584 bytes 2023-12-08T20:40:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 554014720 bytes 2023-12-08T20:40:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 608712704 bytes 2023-12-08T20:40:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 639694848 bytes 2023-12-08T20:40:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 686531584 bytes 2023-12-08T20:40:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 744668160 bytes 2023-12-08T20:40:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 796163072 bytes 2023-12-08T20:40:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 821231616 bytes 2023-12-08T20:40:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 871252992 bytes 2023-12-08T20:40:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 893469696 bytes 2023-12-08T20:40:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 927441920 bytes 2023-12-08T20:40:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 996156416 bytes 2023-12-08T20:40:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1043975168 bytes 2023-12-08T20:40:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1113987072 bytes 2023-12-08T20:40:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1162224640 bytes 2023-12-08T20:40:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1162842112 bytes 2023-12-08T20:40:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1220411392 bytes 2023-12-08T20:40:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1325115392 bytes 2023-12-08T20:40:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1420644352 bytes 2023-12-08T20:40:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1443638272 bytes 2023-12-08T20:40:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1512199168 bytes 2023-12-08T20:40:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1544406016 bytes 2023-12-08T20:40:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1608728576 bytes 2023-12-08T20:40:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1631565824 bytes 2023-12-08T20:40:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1665334272 bytes 2023-12-08T20:40:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1689229312 bytes 2023-12-08T20:40:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1718097920 bytes 2023-12-08T20:40:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1747055616 bytes 2023-12-08T20:40:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1788503040 bytes 2023-12-08T20:40:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1838227456 bytes 2023-12-08T20:40:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1865499648 bytes 2023-12-08T20:40:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1927831552 bytes 2023-12-08T20:40:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1949515776 bytes 2023-12-08T20:40:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1972445184 bytes 2023-12-08T20:40:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2050786304 bytes 2023-12-08T20:40:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2068783104 bytes 2023-12-08T20:40:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2125414400 bytes 2023-12-08T20:40:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2207814656 bytes 2023-12-08T20:40:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:40:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:40:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:40:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_34 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2023-12-08T20:40:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:40:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-08T20:40:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:40:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-08T20:40:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:40:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_34': 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13f350b71b55 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13f350b71b55 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:40:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:31 2023 2023-12-08T20:40:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:31 2023 2023-12-08T20:40:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:31 2023 2023-12-08T20:40:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:40:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:40:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 49f707d8-a6ca-42cf-aebb-9130be1a4ef5 2023-12-08T20:40:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:40:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:40:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-08T20:40:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:40:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:40:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=13 2023-12-08T20:40:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:40:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:40:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 28168192 bytes 2023-12-08T20:40:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 41408512 bytes 2023-12-08T20:40:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 126321664 bytes 2023-12-08T20:40:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 180443136 bytes 2023-12-08T20:40:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 201404416 bytes 2023-12-08T20:40:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 202029056 bytes 2023-12-08T20:40:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 218786816 bytes 2023-12-08T20:40:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 289798144 bytes 2023-12-08T20:40:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 337576960 bytes 2023-12-08T20:40:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 372979712 bytes 2023-12-08T20:40:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 394327040 bytes 2023-12-08T20:40:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 477218816 bytes 2023-12-08T20:40:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 529184768 bytes 2023-12-08T20:40:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 600388608 bytes 2023-12-08T20:40:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 652288000 bytes 2023-12-08T20:40:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 697661440 bytes 2023-12-08T20:40:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 716832768 bytes 2023-12-08T20:40:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 727382016 bytes 2023-12-08T20:40:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 758872064 bytes 2023-12-08T20:40:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 767444992 bytes 2023-12-08T20:40:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 832769024 bytes 2023-12-08T20:40:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 885764096 bytes 2023-12-08T20:40:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 948831232 bytes 2023-12-08T20:40:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 999158784 bytes 2023-12-08T20:40:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1045827584 bytes 2023-12-08T20:40:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1106908160 bytes 2023-12-08T20:40:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1141903360 bytes 2023-12-08T20:40:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1142510592 bytes 2023-12-08T20:40:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1208454144 bytes 2023-12-08T20:40:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1219070976 bytes 2023-12-08T20:40:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1276888064 bytes 2023-12-08T20:40:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1309639680 bytes 2023-12-08T20:40:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1369747456 bytes 2023-12-08T20:40:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1426538496 bytes 2023-12-08T20:40:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1435857920 bytes 2023-12-08T20:40:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1436478464 bytes 2023-12-08T20:40:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1448814592 bytes 2023-12-08T20:40:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1485190144 bytes 2023-12-08T20:40:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1546178560 bytes 2023-12-08T20:40:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1601051648 bytes 2023-12-08T20:40:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1623981056 bytes 2023-12-08T20:40:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1689118720 bytes 2023-12-08T20:40:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1765381120 bytes 2023-12-08T20:40:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1821520896 bytes 2023-12-08T20:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1884440576 bytes 2023-12-08T20:40:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1918021632 bytes 2023-12-08T20:40:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1942044672 bytes 2023-12-08T20:40:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1972265984 bytes 2023-12-08T20:40:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1984631808 bytes 2023-12-08T20:40:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2079128576 bytes 2023-12-08T20:40:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2102126592 bytes 2023-12-08T20:40:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2130478080 bytes 2023-12-08T20:40:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2226332672 bytes 2023-12-08T20:40:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2250359808 bytes 2023-12-08T20:40:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2290826240 bytes 2023-12-08T20:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2429361152 bytes 2023-12-08T20:40:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2433742848 bytes 2023-12-08T20:40:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2538236928 bytes 2023-12-08T20:40:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2548660224 bytes 2023-12-08T20:40:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2585640960 bytes 2023-12-08T20:40:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2671768576 bytes 2023-12-08T20:40:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2748596224 bytes 2023-12-08T20:40:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2818485248 bytes 2023-12-08T20:40:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2829619200 bytes 2023-12-08T20:40:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2854691840 bytes 2023-12-08T20:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2895603712 bytes 2023-12-08T20:40:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2968484864 bytes 2023-12-08T20:40:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3002015744 bytes 2023-12-08T20:40:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3055134720 bytes 2023-12-08T20:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3094586368 bytes 2023-12-08T20:40:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3173109760 bytes 2023-12-08T20:40:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3183376384 bytes 2023-12-08T20:40:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3216971776 bytes 2023-12-08T20:40:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3269680128 bytes 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_35 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2023-12-08T20:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-08T20:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-08T20:40:45.374 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.33408261399432715) > weight (0.25) 2023-12-08T20:40:45.374 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8222291482378655) > weight (0.25) 2023-12-08T20:40:45.374 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8157323895546443) > weight (0.25) 2023-12-08T20:40:45.374 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5397149156932024) > weight (0.25) 2023-12-08T20:40:45.374 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-08T20:40:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:40:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-08T20:40:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_35': 2023-12-08T20:40:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:40:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 13fe13f7573a 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.13fe13f7573a 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:45 2023 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:45 2023 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:45 2023 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 6ff2bf31-0b62-4e55-8ee7-bb0a0e2e75a0 2023-12-08T20:40:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:40:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:40:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-08T20:40:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:40:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:40:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=7 2023-12-08T20:40:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:40:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:40:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 46117888 bytes 2023-12-08T20:40:46.534 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:40:46.520+0000 7fb534865700 -1 reset not still connected to 0x558a29ba52b0 2023-12-08T20:40:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 87114752 bytes 2023-12-08T20:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 116358144 bytes 2023-12-08T20:40:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 151889920 bytes 2023-12-08T20:40:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 171267072 bytes 2023-12-08T20:40:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 244815872 bytes 2023-12-08T20:40:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 352111616 bytes 2023-12-08T20:40:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 368355328 bytes 2023-12-08T20:40:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 382217216 bytes 2023-12-08T20:40:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 452628480 bytes 2023-12-08T20:40:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 463540224 bytes 2023-12-08T20:40:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 481536000 bytes 2023-12-08T20:40:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 529366016 bytes 2023-12-08T20:40:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 577131520 bytes 2023-12-08T20:40:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 587577344 bytes 2023-12-08T20:40:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 615286784 bytes 2023-12-08T20:40:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 689285120 bytes 2023-12-08T20:40:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 727161856 bytes 2023-12-08T20:40:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 848979968 bytes 2023-12-08T20:40:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 929701888 bytes 2023-12-08T20:40:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 968443904 bytes 2023-12-08T20:40:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1006467072 bytes 2023-12-08T20:40:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1030328320 bytes 2023-12-08T20:40:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1083220992 bytes 2023-12-08T20:40:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1142718464 bytes 2023-12-08T20:40:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1164884992 bytes 2023-12-08T20:40:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1197324288 bytes 2023-12-08T20:40:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1277712384 bytes 2023-12-08T20:40:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1290374144 bytes 2023-12-08T20:40:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1307142144 bytes 2023-12-08T20:40:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1376310272 bytes 2023-12-08T20:40:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1398255616 bytes 2023-12-08T20:40:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1406580736 bytes 2023-12-08T20:40:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1430646784 bytes 2023-12-08T20:40:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1452372992 bytes 2023-12-08T20:40:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1494407168 bytes 2023-12-08T20:40:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1576841216 bytes 2023-12-08T20:40:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1634818048 bytes 2023-12-08T20:40:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1638829056 bytes 2023-12-08T20:40:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:40:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:40:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_36 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-08T20:40:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2023-12-08T20:40:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:40:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-08T20:40:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:40:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-08T20:40:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-08T20:40:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_36': 2023-12-08T20:40:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:40:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:40:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:40:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14074ca4d5ae 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14074ca4d5ae 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:53 2023 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:53 2023 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:53 2023 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 4be21f09-5358-40e3-8098-0e9b567e3808 2023-12-08T20:40:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:40:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:40:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-08T20:40:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:40:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:40:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=5 2023-12-08T20:40:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:40:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:40:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 20931584 bytes 2023-12-08T20:40:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 45812736 bytes 2023-12-08T20:40:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 115683328 bytes 2023-12-08T20:40:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 184048640 bytes 2023-12-08T20:40:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 249106432 bytes 2023-12-08T20:40:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 288927744 bytes 2023-12-08T20:40:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 322975744 bytes 2023-12-08T20:40:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 421591040 bytes 2023-12-08T20:40:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 441918464 bytes 2023-12-08T20:40:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 521958400 bytes 2023-12-08T20:40:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 651595776 bytes 2023-12-08T20:40:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 660221952 bytes 2023-12-08T20:40:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 660813824 bytes 2023-12-08T20:40:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 709908480 bytes 2023-12-08T20:40:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 760617984 bytes 2023-12-08T20:40:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 782692352 bytes 2023-12-08T20:40:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 812101632 bytes 2023-12-08T20:40:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 840350720 bytes 2023-12-08T20:40:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 856600576 bytes 2023-12-08T20:40:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 892803072 bytes 2023-12-08T20:40:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 946072576 bytes 2023-12-08T20:40:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1040854016 bytes 2023-12-08T20:40:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1090276352 bytes 2023-12-08T20:40:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1135450112 bytes 2023-12-08T20:40:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1202632704 bytes 2023-12-08T20:40:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1211668480 bytes 2023-12-08T20:40:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1274945536 bytes 2023-12-08T20:40:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1306390528 bytes 2023-12-08T20:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_37 2023-12-08T20:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-08T20:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2023-12-08T20:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-08T20:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-08T20:40:59.475 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.37740697824372726) > weight (0.25) 2023-12-08T20:40:59.475 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.39825126065356586) > weight (0.25) 2023-12-08T20:40:59.476 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-08T20:40:59.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-08T20:40:59.476 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-08T20:40:59.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:40:59.468+0000 7f23e8a45700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 43186) UID: 1000 2023-12-08T20:40:59.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:41:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:41:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-08T20:41:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_37': 2023-12-08T20:41:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14134f082466 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14134f082466 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:59 2023 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:59 2023 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:59 2023 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 614d01e3-7add-4c70-abcb-94fcdd953061 2023-12-08T20:41:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:41:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:41:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-08T20:41:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:41:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:41:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=14 2023-12-08T20:41:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:41:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:41:00.477 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-08T20:41:00.477 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:41:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:41:00.477 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-08T20:41:00.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-08T20:41:00.478 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-08T20:41:00.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-08T20:41:00.480 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-08T20:41:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 8577024 bytes 2023-12-08T20:41:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 88705024 bytes 2023-12-08T20:41:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 106740736 bytes 2023-12-08T20:41:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 117111808 bytes 2023-12-08T20:41:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 165715968 bytes 2023-12-08T20:41:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 216637440 bytes 2023-12-08T20:41:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 273615872 bytes 2023-12-08T20:41:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 281982976 bytes 2023-12-08T20:41:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 290281472 bytes 2023-12-08T20:41:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 352215040 bytes 2023-12-08T20:41:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 432006144 bytes 2023-12-08T20:41:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 479519744 bytes 2023-12-08T20:41:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 529799168 bytes 2023-12-08T20:41:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 602786816 bytes 2023-12-08T20:41:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 663114752 bytes 2023-12-08T20:41:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 733982720 bytes 2023-12-08T20:41:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 786757632 bytes 2023-12-08T20:41:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 799297536 bytes 2023-12-08T20:41:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 859245568 bytes 2023-12-08T20:41:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 892045312 bytes 2023-12-08T20:41:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 903142400 bytes 2023-12-08T20:41:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 940892160 bytes 2023-12-08T20:41:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1057505280 bytes 2023-12-08T20:41:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1075332096 bytes 2023-12-08T20:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1132293120 bytes 2023-12-08T20:41:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1174500352 bytes 2023-12-08T20:41:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1203846144 bytes 2023-12-08T20:41:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1204460544 bytes 2023-12-08T20:41:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1234860032 bytes 2023-12-08T20:41:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1291948032 bytes 2023-12-08T20:41:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1333808128 bytes 2023-12-08T20:41:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1389365248 bytes 2023-12-08T20:41:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1432007680 bytes 2023-12-08T20:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1461102592 bytes 2023-12-08T20:41:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1523646464 bytes 2023-12-08T20:41:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1556296704 bytes 2023-12-08T20:41:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1601226752 bytes 2023-12-08T20:41:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1629159424 bytes 2023-12-08T20:41:07.081 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1648195584 bytes 2023-12-08T20:41:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1705453568 bytes 2023-12-08T20:41:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1725923328 bytes 2023-12-08T20:41:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1795469312 bytes 2023-12-08T20:41:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1857289216 bytes 2023-12-08T20:41:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1870380032 bytes 2023-12-08T20:41:08.090 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:08.080+0000 7f587a51c700 -1 reset not still connected to 0x55bc444aa820 2023-12-08T20:41:08.095 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:08.084+0000 7f587a51c700 -1 reset not still connected to 0x55bc444aab60 2023-12-08T20:41:08.095 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:08.084+0000 7f587a51c700 -1 reset not still connected to 0x55bc44b84820 2023-12-08T20:41:08.099 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:08.088+0000 7f587a51c700 -1 reset not still connected to 0x55bc44b85110 2023-12-08T20:41:08.107 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:08.096+0000 7f587a51c700 -1 reset not still connected to 0x55bc44b856c0 2023-12-08T20:41:08.107 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:08.096+0000 7f587a51c700 -1 reset not still connected to 0x55bc4f640340 2023-12-08T20:41:08.112 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:08.100+0000 7f587a51c700 -1 reset not still connected to 0x55bc54ea5790 2023-12-08T20:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1952398336 bytes 2023-12-08T20:41:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1956679680 bytes 2023-12-08T20:41:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1988327424 bytes 2023-12-08T20:41:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2010437632 bytes 2023-12-08T20:41:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2157886464 bytes 2023-12-08T20:41:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2208030720 bytes 2023-12-08T20:41:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2273931264 bytes 2023-12-08T20:41:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2308142080 bytes 2023-12-08T20:41:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2335230976 bytes 2023-12-08T20:41:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2335846400 bytes 2023-12-08T20:41:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2343611392 bytes 2023-12-08T20:41:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2367041536 bytes 2023-12-08T20:41:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2409430016 bytes 2023-12-08T20:41:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2439794688 bytes 2023-12-08T20:41:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2461892608 bytes 2023-12-08T20:41:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2494947328 bytes 2023-12-08T20:41:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2536235008 bytes 2023-12-08T20:41:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2580904960 bytes 2023-12-08T20:41:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2604656640 bytes 2023-12-08T20:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2644404224 bytes 2023-12-08T20:41:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2684022784 bytes 2023-12-08T20:41:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2721729536 bytes 2023-12-08T20:41:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2729382912 bytes 2023-12-08T20:41:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2747838464 bytes 2023-12-08T20:41:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 2776102912 bytes 2023-12-08T20:41:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 2837799936 bytes 2023-12-08T20:41:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 2854323200 bytes 2023-12-08T20:41:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 2858624000 bytes 2023-12-08T20:41:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 2916869120 bytes 2023-12-08T20:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 2958738432 bytes 2023-12-08T20:41:13.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 2985464832 bytes 2023-12-08T20:41:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3005065216 bytes 2023-12-08T20:41:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3025296384 bytes 2023-12-08T20:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3031263232 bytes 2023-12-08T20:41:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3076801536 bytes 2023-12-08T20:41:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8000 IOs, 3106904064 bytes 2023-12-08T20:41:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8100 IOs, 3153806336 bytes 2023-12-08T20:41:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8200 IOs, 3207198720 bytes 2023-12-08T20:41:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:8300 IOs, 3207829504 bytes 2023-12-08T20:41:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:41:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_38 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:41:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2023-12-08T20:41:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:41:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-08T20:41:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:41:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-08T20:41:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-08T20:41:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_38': 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14225b1f9a1f 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14225b1f9a1f 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:41:14 2023 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:41:14 2023 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:41:14 2023 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:41:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:41:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 2b65f3dc-bcb4-4889-8f5a-69e9969de719 2023-12-08T20:41:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:41:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:41:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-08T20:41:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:41:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:41:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=11 2023-12-08T20:41:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:41:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:41:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 55497728 bytes 2023-12-08T20:41:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 131981312 bytes 2023-12-08T20:41:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 230042624 bytes 2023-12-08T20:41:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 273839104 bytes 2023-12-08T20:41:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 309275648 bytes 2023-12-08T20:41:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 355107840 bytes 2023-12-08T20:41:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 414713856 bytes 2023-12-08T20:41:16.769 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 427735040 bytes 2023-12-08T20:41:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 455060480 bytes 2023-12-08T20:41:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 476692480 bytes 2023-12-08T20:41:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 513580032 bytes 2023-12-08T20:41:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 548080640 bytes 2023-12-08T20:41:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 591953920 bytes 2023-12-08T20:41:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 621227008 bytes 2023-12-08T20:41:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 739608576 bytes 2023-12-08T20:41:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 781020160 bytes 2023-12-08T20:41:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 822790144 bytes 2023-12-08T20:41:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 838320128 bytes 2023-12-08T20:41:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 897775616 bytes 2023-12-08T20:41:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 921847808 bytes 2023-12-08T20:41:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 975779840 bytes 2023-12-08T20:41:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1026960384 bytes 2023-12-08T20:41:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1047544832 bytes 2023-12-08T20:41:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1089784832 bytes 2023-12-08T20:41:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1134491648 bytes 2023-12-08T20:41:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1161638912 bytes 2023-12-08T20:41:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1171378176 bytes 2023-12-08T20:41:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1196140544 bytes 2023-12-08T20:41:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1196772352 bytes 2023-12-08T20:41:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1228937216 bytes 2023-12-08T20:41:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1231793152 bytes 2023-12-08T20:41:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1238841344 bytes 2023-12-08T20:41:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1253482496 bytes 2023-12-08T20:41:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1324960768 bytes 2023-12-08T20:41:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1402184704 bytes 2023-12-08T20:41:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1483220992 bytes 2023-12-08T20:41:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1560883200 bytes 2023-12-08T20:41:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1596926976 bytes 2023-12-08T20:41:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1604269056 bytes 2023-12-08T20:41:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1643603968 bytes 2023-12-08T20:41:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1694307328 bytes 2023-12-08T20:41:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1700531200 bytes 2023-12-08T20:41:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1716817920 bytes 2023-12-08T20:41:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1752122368 bytes 2023-12-08T20:41:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1800232960 bytes 2023-12-08T20:41:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1819622400 bytes 2023-12-08T20:41:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1820233728 bytes 2023-12-08T20:41:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1859874816 bytes 2023-12-08T20:41:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1877981184 bytes 2023-12-08T20:41:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 1883942912 bytes 2023-12-08T20:41:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 1942895616 bytes 2023-12-08T20:41:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2006778880 bytes 2023-12-08T20:41:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2018827264 bytes 2023-12-08T20:41:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2056957952 bytes 2023-12-08T20:41:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2104150016 bytes 2023-12-08T20:41:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2189538304 bytes 2023-12-08T20:41:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2219875328 bytes 2023-12-08T20:41:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2275014656 bytes 2023-12-08T20:41:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2326233088 bytes 2023-12-08T20:41:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2367681536 bytes 2023-12-08T20:41:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2392355840 bytes 2023-12-08T20:41:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2457795584 bytes 2023-12-08T20:41:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2502712320 bytes 2023-12-08T20:41:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_39 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2023-12-08T20:41:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:41:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-08T20:41:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:41:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-08T20:41:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:41:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-08T20:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_39': 2023-12-08T20:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 142a7642049c 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.142a7642049c 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:41:26 2023 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:41:26 2023 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:41:26 2023 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 9997775c-0583-4526-b5bc-331b9b32f090 2023-12-08T20:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:41:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:41:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-08T20:41:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:41:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:41:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=11 2023-12-08T20:41:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:41:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:41:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 72142848 bytes 2023-12-08T20:41:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 72729600 bytes 2023-12-08T20:41:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 98451456 bytes 2023-12-08T20:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 153935872 bytes 2023-12-08T20:41:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 209842176 bytes 2023-12-08T20:41:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 234271744 bytes 2023-12-08T20:41:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 315245568 bytes 2023-12-08T20:41:28.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 350876672 bytes 2023-12-08T20:41:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 377584640 bytes 2023-12-08T20:41:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 461639680 bytes 2023-12-08T20:41:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 497513472 bytes 2023-12-08T20:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 548654080 bytes 2023-12-08T20:41:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 604108800 bytes 2023-12-08T20:41:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 628237312 bytes 2023-12-08T20:41:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 693425152 bytes 2023-12-08T20:41:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 713966592 bytes 2023-12-08T20:41:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 762722304 bytes 2023-12-08T20:41:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 804058112 bytes 2023-12-08T20:41:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 804656128 bytes 2023-12-08T20:41:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 872060928 bytes 2023-12-08T20:41:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1021558784 bytes 2023-12-08T20:41:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1034248192 bytes 2023-12-08T20:41:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1099553792 bytes 2023-12-08T20:41:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1137819648 bytes 2023-12-08T20:41:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1191569408 bytes 2023-12-08T20:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1234094080 bytes 2023-12-08T20:41:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1314469888 bytes 2023-12-08T20:41:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1361618944 bytes 2023-12-08T20:41:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1373455360 bytes 2023-12-08T20:41:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1461232640 bytes 2023-12-08T20:41:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1529668608 bytes 2023-12-08T20:41:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1581344768 bytes 2023-12-08T20:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1615301632 bytes 2023-12-08T20:41:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1636504576 bytes 2023-12-08T20:41:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1640290304 bytes 2023-12-08T20:41:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1694536704 bytes 2023-12-08T20:41:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1793815552 bytes 2023-12-08T20:41:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1896102912 bytes 2023-12-08T20:41:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1916235776 bytes 2023-12-08T20:41:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1990216704 bytes 2023-12-08T20:41:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 2057059328 bytes 2023-12-08T20:41:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 2077806592 bytes 2023-12-08T20:41:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2116334592 bytes 2023-12-08T20:41:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2146440192 bytes 2023-12-08T20:41:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2180077568 bytes 2023-12-08T20:41:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2245768192 bytes 2023-12-08T20:41:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2333093888 bytes 2023-12-08T20:41:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2376318976 bytes 2023-12-08T20:41:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2447883264 bytes 2023-12-08T20:41:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2487949312 bytes 2023-12-08T20:41:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2508364800 bytes 2023-12-08T20:41:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2549629952 bytes 2023-12-08T20:41:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2587625472 bytes 2023-12-08T20:41:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2655612928 bytes 2023-12-08T20:41:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2704508928 bytes 2023-12-08T20:41:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2735242240 bytes 2023-12-08T20:41:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2762490880 bytes 2023-12-08T20:41:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2792918016 bytes 2023-12-08T20:41:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2842248192 bytes 2023-12-08T20:41:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2842857472 bytes 2023-12-08T20:41:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2898676736 bytes 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_40 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_40 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_40 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:41:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:41:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2023-12-08T20:41:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:41:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-08T20:41:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_40 2023-12-08T20:41:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-08T20:41:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:41:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-08T20:41:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_40': 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14333618e2c9 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14333618e2c9 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:41:38 2023 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:41:38 2023 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:41:38 2023 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bc088a6c-7d30-45a7-8484-a67f43a12e02 2023-12-08T20:41:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:41:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:41:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-08T20:41:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_40 2023-12-08T20:41:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:41:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=8 2023-12-08T20:41:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:41:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:41:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 67080192 bytes 2023-12-08T20:41:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 102319104 bytes 2023-12-08T20:41:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 118020096 bytes 2023-12-08T20:41:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 131395584 bytes 2023-12-08T20:41:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 177046528 bytes 2023-12-08T20:41:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 200945664 bytes 2023-12-08T20:41:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 240997376 bytes 2023-12-08T20:41:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 280966144 bytes 2023-12-08T20:41:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 335474688 bytes 2023-12-08T20:41:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 435855360 bytes 2023-12-08T20:41:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 593594368 bytes 2023-12-08T20:41:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 601792512 bytes 2023-12-08T20:41:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 631851008 bytes 2023-12-08T20:41:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 705215488 bytes 2023-12-08T20:41:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 792326144 bytes 2023-12-08T20:41:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 828963840 bytes 2023-12-08T20:41:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 861452288 bytes 2023-12-08T20:41:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 881152000 bytes 2023-12-08T20:41:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 889718784 bytes 2023-12-08T20:41:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 936247296 bytes 2023-12-08T20:41:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 948238336 bytes 2023-12-08T20:41:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1019164672 bytes 2023-12-08T20:41:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1081545728 bytes 2023-12-08T20:41:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1182652416 bytes 2023-12-08T20:41:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1205842944 bytes 2023-12-08T20:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1258927104 bytes 2023-12-08T20:41:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1300328448 bytes 2023-12-08T20:41:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1315936256 bytes 2023-12-08T20:41:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1387713536 bytes 2023-12-08T20:41:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1390081024 bytes 2023-12-08T20:41:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1452267520 bytes 2023-12-08T20:41:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1535997952 bytes 2023-12-08T20:41:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1576795136 bytes 2023-12-08T20:41:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1582556160 bytes 2023-12-08T20:41:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1614394368 bytes 2023-12-08T20:41:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1620408320 bytes 2023-12-08T20:41:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1680733184 bytes 2023-12-08T20:41:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1727298560 bytes 2023-12-08T20:41:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1727890432 bytes 2023-12-08T20:41:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1763859456 bytes 2023-12-08T20:41:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1838650368 bytes 2023-12-08T20:41:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1881170944 bytes 2023-12-08T20:41:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1893505024 bytes 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_41 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_41 2023-12-08T20:41:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_41 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:41:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:41:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2023-12-08T20:41:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:41:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-08T20:41:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_41 2023-12-08T20:41:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-08T20:41:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:41:48.126 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:48.112+0000 7f587a51c700 -1 reset not still connected to 0x55bc44b856c0 2023-12-08T20:41:48.126 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:48.112+0000 7f587a51c700 -1 reset not still connected to 0x55bc4f640340 2023-12-08T20:41:48.130 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:41:48.120+0000 7f587a51c700 -1 reset not still connected to 0x55bc54ea5790 2023-12-08T20:41:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_41': 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 143c7e9fd6fa 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.143c7e9fd6fa 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:41:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:41:47 2023 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:41:47 2023 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:41:47 2023 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: d18a2bfd-3fe4-41dc-911a-4b6bba543bcf 2023-12-08T20:41:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:41:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:41:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-08T20:41:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_41 2023-12-08T20:41:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:41:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=11 2023-12-08T20:41:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:41:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:41:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 57768960 bytes 2023-12-08T20:41:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 97964032 bytes 2023-12-08T20:41:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 219316224 bytes 2023-12-08T20:41:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 223663104 bytes 2023-12-08T20:41:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 271495168 bytes 2023-12-08T20:41:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 309667840 bytes 2023-12-08T20:41:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 404891648 bytes 2023-12-08T20:41:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 461678592 bytes 2023-12-08T20:41:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 504627200 bytes 2023-12-08T20:41:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 552254464 bytes 2023-12-08T20:41:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 561639424 bytes 2023-12-08T20:41:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 633401344 bytes 2023-12-08T20:41:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 641375232 bytes 2023-12-08T20:41:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 673438720 bytes 2023-12-08T20:41:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 739478528 bytes 2023-12-08T20:41:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 758064128 bytes 2023-12-08T20:41:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 761640960 bytes 2023-12-08T20:41:51.421 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 784150528 bytes 2023-12-08T20:41:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 850035712 bytes 2023-12-08T20:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 850669568 bytes 2023-12-08T20:41:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 881387520 bytes 2023-12-08T20:41:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 909436928 bytes 2023-12-08T20:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 937563136 bytes 2023-12-08T20:41:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1007376384 bytes 2023-12-08T20:41:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1056743424 bytes 2023-12-08T20:41:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1075407872 bytes 2023-12-08T20:41:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1104623616 bytes 2023-12-08T20:41:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1149987840 bytes 2023-12-08T20:41:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1173864448 bytes 2023-12-08T20:41:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1233891328 bytes 2023-12-08T20:41:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1256161280 bytes 2023-12-08T20:41:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1311045632 bytes 2023-12-08T20:41:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1333929984 bytes 2023-12-08T20:41:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1359844352 bytes 2023-12-08T20:41:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1380477952 bytes 2023-12-08T20:41:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1430384640 bytes 2023-12-08T20:41:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1458085888 bytes 2023-12-08T20:41:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1490480128 bytes 2023-12-08T20:41:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1570253824 bytes 2023-12-08T20:41:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1583673344 bytes 2023-12-08T20:41:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1632200704 bytes 2023-12-08T20:41:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1684111360 bytes 2023-12-08T20:41:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1764491264 bytes 2023-12-08T20:41:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1871424512 bytes 2023-12-08T20:41:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1925807104 bytes 2023-12-08T20:41:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1948702720 bytes 2023-12-08T20:41:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1968197632 bytes 2023-12-08T20:41:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2024512512 bytes 2023-12-08T20:41:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2061590528 bytes 2023-12-08T20:41:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2068609024 bytes 2023-12-08T20:41:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2106354688 bytes 2023-12-08T20:41:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2147953664 bytes 2023-12-08T20:41:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2148589568 bytes 2023-12-08T20:41:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2204813312 bytes 2023-12-08T20:41:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2205426688 bytes 2023-12-08T20:41:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2287768576 bytes 2023-12-08T20:41:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2357708800 bytes 2023-12-08T20:41:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2377905152 bytes 2023-12-08T20:41:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2412057600 bytes 2023-12-08T20:41:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2431328256 bytes 2023-12-08T20:41:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2456200192 bytes 2023-12-08T20:41:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2501748736 bytes 2023-12-08T20:41:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2526910464 bytes 2023-12-08T20:41:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2540499968 bytes 2023-12-08T20:41:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2594388992 bytes 2023-12-08T20:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_42 2023-12-08T20:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-08T20:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_42 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_42 2023-12-08T20:41:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:41:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:41:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:41:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:41:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2023-12-08T20:41:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:41:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-08T20:41:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:41:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:41:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_42 2023-12-08T20:41:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:41:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-08T20:42:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:42:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-08T20:42:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_42': 2023-12-08T20:42:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:42:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:42:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14454c4c1380 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14454c4c1380 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:41:59 2023 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:41:59 2023 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:41:59 2023 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 20981ed9-cf2d-4172-8916-f3d449ef5ee9 2023-12-08T20:42:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:42:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:42:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-08T20:42:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_42 2023-12-08T20:42:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:42:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=14 2023-12-08T20:42:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:42:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:42:00.375 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.43728296438811143) > weight (0.25) 2023-12-08T20:42:00.375 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.36815561750803305) > weight (0.25) 2023-12-08T20:42:00.376 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-08T20:42:00.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-08T20:42:00.376 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-08T20:42:00.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi158.stderr:2023-12-08T20:42:00.368+0000 7ff19d05f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 37365) UID: 1000 2023-12-08T20:42:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 48782336 bytes 2023-12-08T20:42:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 63531008 bytes 2023-12-08T20:42:00.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:42:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 120285184 bytes 2023-12-08T20:42:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 176590848 bytes 2023-12-08T20:42:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 194447360 bytes 2023-12-08T20:42:01.377 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-08T20:42:01.378 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:42:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:42:01.378 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-08T20:42:01.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-08T20:42:01.378 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-08T20:42:01.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-08T20:42:01.381 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-08T20:42:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 224539648 bytes 2023-12-08T20:42:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 249281536 bytes 2023-12-08T20:42:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 270449664 bytes 2023-12-08T20:42:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 310799360 bytes 2023-12-08T20:42:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 338633728 bytes 2023-12-08T20:42:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 398125056 bytes 2023-12-08T20:42:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 495376384 bytes 2023-12-08T20:42:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 514330624 bytes 2023-12-08T20:42:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 517531648 bytes 2023-12-08T20:42:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 588695552 bytes 2023-12-08T20:42:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 632371200 bytes 2023-12-08T20:42:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 704535552 bytes 2023-12-08T20:42:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 808497152 bytes 2023-12-08T20:42:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 851158016 bytes 2023-12-08T20:42:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 906148864 bytes 2023-12-08T20:42:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 943722496 bytes 2023-12-08T20:42:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 977467392 bytes 2023-12-08T20:42:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1035830272 bytes 2023-12-08T20:42:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1066647552 bytes 2023-12-08T20:42:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1092358144 bytes 2023-12-08T20:42:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1109483520 bytes 2023-12-08T20:42:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1144478720 bytes 2023-12-08T20:42:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1206778880 bytes 2023-12-08T20:42:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1256031232 bytes 2023-12-08T20:42:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1276328960 bytes 2023-12-08T20:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1293455360 bytes 2023-12-08T20:42:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1318573056 bytes 2023-12-08T20:42:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1329143808 bytes 2023-12-08T20:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1359017984 bytes 2023-12-08T20:42:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1428800512 bytes 2023-12-08T20:42:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1487372288 bytes 2023-12-08T20:42:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1549752320 bytes 2023-12-08T20:42:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1614906368 bytes 2023-12-08T20:42:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1640035328 bytes 2023-12-08T20:42:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1655552000 bytes 2023-12-08T20:42:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1699217408 bytes 2023-12-08T20:42:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1719653376 bytes 2023-12-08T20:42:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1730944000 bytes 2023-12-08T20:42:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1778378752 bytes 2023-12-08T20:42:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1897129984 bytes 2023-12-08T20:42:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1924803584 bytes 2023-12-08T20:42:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1946498048 bytes 2023-12-08T20:42:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1998173184 bytes 2023-12-08T20:42:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2050648064 bytes 2023-12-08T20:42:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2129492992 bytes 2023-12-08T20:42:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2180444160 bytes 2023-12-08T20:42:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2254714880 bytes 2023-12-08T20:42:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2317059072 bytes 2023-12-08T20:42:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2348688384 bytes 2023-12-08T20:42:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2397074432 bytes 2023-12-08T20:42:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2415882240 bytes 2023-12-08T20:42:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2519408640 bytes 2023-12-08T20:42:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2550281216 bytes 2023-12-08T20:42:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2604092416 bytes 2023-12-08T20:42:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2684669952 bytes 2023-12-08T20:42:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2759825408 bytes 2023-12-08T20:42:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2843712512 bytes 2023-12-08T20:42:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2902779904 bytes 2023-12-08T20:42:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2980059136 bytes 2023-12-08T20:42:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 3011662848 bytes 2023-12-08T20:42:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 3051698176 bytes 2023-12-08T20:42:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 3080084480 bytes 2023-12-08T20:42:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 3101749248 bytes 2023-12-08T20:42:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 3153357824 bytes 2023-12-08T20:42:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 3183488000 bytes 2023-12-08T20:42:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 3240577024 bytes 2023-12-08T20:42:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 3296451584 bytes 2023-12-08T20:42:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3322648576 bytes 2023-12-08T20:42:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3362069504 bytes 2023-12-08T20:42:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3391851520 bytes 2023-12-08T20:42:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3412794368 bytes 2023-12-08T20:42:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3478651904 bytes 2023-12-08T20:42:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:42:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:42:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:42:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:42:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_43 2023-12-08T20:42:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_43 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_43 2023-12-08T20:42:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:42:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:42:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:42:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2023-12-08T20:42:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:42:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-08T20:42:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_43 2023-12-08T20:42:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-08T20:42:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:42:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-08T20:42:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_43': 2023-12-08T20:42:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:42:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:42:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:42:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1458e937e59d 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1458e937e59d 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:42:14 2023 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:42:14 2023 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:42:14 2023 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 006b0561-323e-4305-a2ce-53ff64ba992f 2023-12-08T20:42:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_43 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=7 2023-12-08T20:42:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:42:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:42:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 42014720 bytes 2023-12-08T20:42:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 64461824 bytes 2023-12-08T20:42:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 110038016 bytes 2023-12-08T20:42:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 204779520 bytes 2023-12-08T20:42:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 224406528 bytes 2023-12-08T20:42:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 314573824 bytes 2023-12-08T20:42:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 386903040 bytes 2023-12-08T20:42:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 406109184 bytes 2023-12-08T20:42:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 464340992 bytes 2023-12-08T20:42:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 472138752 bytes 2023-12-08T20:42:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 555748352 bytes 2023-12-08T20:42:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 581694464 bytes 2023-12-08T20:42:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 653346816 bytes 2023-12-08T20:42:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 693194752 bytes 2023-12-08T20:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 719648768 bytes 2023-12-08T20:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 757722112 bytes 2023-12-08T20:42:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 893132800 bytes 2023-12-08T20:42:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 1010693120 bytes 2023-12-08T20:42:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 1078377472 bytes 2023-12-08T20:42:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 1093493760 bytes 2023-12-08T20:42:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 1094137856 bytes 2023-12-08T20:42:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1141762048 bytes 2023-12-08T20:42:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1156952064 bytes 2023-12-08T20:42:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1177731072 bytes 2023-12-08T20:42:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1192920064 bytes 2023-12-08T20:42:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1221989376 bytes 2023-12-08T20:42:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1238201344 bytes 2023-12-08T20:42:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1326221312 bytes 2023-12-08T20:42:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1348972544 bytes 2023-12-08T20:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1364207616 bytes 2023-12-08T20:42:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1378151424 bytes 2023-12-08T20:42:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1411736576 bytes 2023-12-08T20:42:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1439766528 bytes 2023-12-08T20:42:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1487166464 bytes 2023-12-08T20:42:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1566408704 bytes 2023-12-08T20:42:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1594805248 bytes 2023-12-08T20:42:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1611104256 bytes 2023-12-08T20:42:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1704465408 bytes 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_44 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_44 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_44 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2023-12-08T20:42:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:42:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-08T20:42:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_44 2023-12-08T20:42:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-08T20:42:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:42:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-08T20:42:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_44': 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14601dc01c72 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14601dc01c72 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:42:22 2023 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:42:22 2023 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:42:22 2023 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:42:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:42:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0f77f075-0312-4c92-b2f0-eaedecb9b9ef 2023-12-08T20:42:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:42:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:42:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-08T20:42:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_44 2023-12-08T20:42:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:42:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=7 2023-12-08T20:42:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:42:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:42:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 80819200 bytes 2023-12-08T20:42:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 115566592 bytes 2023-12-08T20:42:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 155119616 bytes 2023-12-08T20:42:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 197215232 bytes 2023-12-08T20:42:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 226750464 bytes 2023-12-08T20:42:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 233075712 bytes 2023-12-08T20:42:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 257108992 bytes 2023-12-08T20:42:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 342711296 bytes 2023-12-08T20:42:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 371798016 bytes 2023-12-08T20:42:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 375096320 bytes 2023-12-08T20:42:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 423372800 bytes 2023-12-08T20:42:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 463251456 bytes 2023-12-08T20:42:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 487142400 bytes 2023-12-08T20:42:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 515899392 bytes 2023-12-08T20:42:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 551977984 bytes 2023-12-08T20:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 648411136 bytes 2023-12-08T20:42:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 649049088 bytes 2023-12-08T20:42:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 768521216 bytes 2023-12-08T20:42:26.575 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:42:26.564+0000 7f587a51c700 -1 reset not still connected to 0x55bc42fe3ad0 2023-12-08T20:42:26.578 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:42:26.568+0000 7f587a51c700 -1 reset not still connected to 0x55bc436d1790 2023-12-08T20:42:26.579 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:42:26.568+0000 7f587a51c700 -1 reset not still connected to 0x55bc444ab110 2023-12-08T20:42:26.581 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:42:26.572+0000 7f587a51c700 -1 reset not still connected to 0x55bc5740ac30 2023-12-08T20:42:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 803148800 bytes 2023-12-08T20:42:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 928011264 bytes 2023-12-08T20:42:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 968969216 bytes 2023-12-08T20:42:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1014916096 bytes 2023-12-08T20:42:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1023000576 bytes 2023-12-08T20:42:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1075055616 bytes 2023-12-08T20:42:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1124791296 bytes 2023-12-08T20:42:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1144153088 bytes 2023-12-08T20:42:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1198702592 bytes 2023-12-08T20:42:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1233424384 bytes 2023-12-08T20:42:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1266253824 bytes 2023-12-08T20:42:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1310913536 bytes 2023-12-08T20:42:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1359598592 bytes 2023-12-08T20:42:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1398441984 bytes 2023-12-08T20:42:29.114 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1432476672 bytes 2023-12-08T20:42:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1460741120 bytes 2023-12-08T20:42:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1535648768 bytes 2023-12-08T20:42:29.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1585269760 bytes 2023-12-08T20:42:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1620706304 bytes 2023-12-08T20:42:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1679990784 bytes 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_45 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_45 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_45 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2023-12-08T20:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-08T20:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_45 2023-12-08T20:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-08T20:42:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:42:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-08T20:42:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_45': 2023-12-08T20:42:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:42:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:42:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:42:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1469b91dc779 2023-12-08T20:42:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1469b91dc779 2023-12-08T20:42:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:42:30 2023 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:42:30 2023 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:42:30 2023 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 14564ad3-ff83-4fb5-ba38-c42cd4770830 2023-12-08T20:42:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:42:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:42:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-08T20:42:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_45 2023-12-08T20:42:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:42:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=14 2023-12-08T20:42:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:42:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:42:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 6793216 bytes 2023-12-08T20:42:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 18216960 bytes 2023-12-08T20:42:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 111204352 bytes 2023-12-08T20:42:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 237964288 bytes 2023-12-08T20:42:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 267486208 bytes 2023-12-08T20:42:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 290031616 bytes 2023-12-08T20:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 290629632 bytes 2023-12-08T20:42:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 342663168 bytes 2023-12-08T20:42:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 361832448 bytes 2023-12-08T20:42:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 373722112 bytes 2023-12-08T20:42:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 433105920 bytes 2023-12-08T20:42:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 478347264 bytes 2023-12-08T20:42:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 507222016 bytes 2023-12-08T20:42:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 558030848 bytes 2023-12-08T20:42:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 576534528 bytes 2023-12-08T20:42:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 577132544 bytes 2023-12-08T20:42:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 639127552 bytes 2023-12-08T20:42:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 749230080 bytes 2023-12-08T20:42:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 761675776 bytes 2023-12-08T20:42:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 822604800 bytes 2023-12-08T20:42:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 869445632 bytes 2023-12-08T20:42:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 885874688 bytes 2023-12-08T20:42:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 903242752 bytes 2023-12-08T20:42:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 953859072 bytes 2023-12-08T20:42:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1004341248 bytes 2023-12-08T20:42:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1007566848 bytes 2023-12-08T20:42:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1038823424 bytes 2023-12-08T20:42:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1064269824 bytes 2023-12-08T20:42:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1108385792 bytes 2023-12-08T20:42:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1142846464 bytes 2023-12-08T20:42:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1227633664 bytes 2023-12-08T20:42:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1228269568 bytes 2023-12-08T20:42:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1314030592 bytes 2023-12-08T20:42:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1394241536 bytes 2023-12-08T20:42:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1410740224 bytes 2023-12-08T20:42:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1458437120 bytes 2023-12-08T20:42:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1496572928 bytes 2023-12-08T20:42:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1535501312 bytes 2023-12-08T20:42:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1581540352 bytes 2023-12-08T20:42:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1622776832 bytes 2023-12-08T20:42:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1661370368 bytes 2023-12-08T20:42:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1696417792 bytes 2023-12-08T20:42:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1717269504 bytes 2023-12-08T20:42:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1786540032 bytes 2023-12-08T20:42:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1788527616 bytes 2023-12-08T20:42:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1811867648 bytes 2023-12-08T20:42:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1891841024 bytes 2023-12-08T20:42:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1914933248 bytes 2023-12-08T20:42:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1982267392 bytes 2023-12-08T20:42:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2059789312 bytes 2023-12-08T20:42:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2077314048 bytes 2023-12-08T20:42:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2098589696 bytes 2023-12-08T20:42:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2132377600 bytes 2023-12-08T20:42:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2151336960 bytes 2023-12-08T20:42:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2188561408 bytes 2023-12-08T20:42:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2205313024 bytes 2023-12-08T20:42:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2241503232 bytes 2023-12-08T20:42:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2285034496 bytes 2023-12-08T20:42:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2317847552 bytes 2023-12-08T20:42:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2403860480 bytes 2023-12-08T20:42:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2450524160 bytes 2023-12-08T20:42:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2502204416 bytes 2023-12-08T20:42:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2502792192 bytes 2023-12-08T20:42:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2536249344 bytes 2023-12-08T20:42:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2600801280 bytes 2023-12-08T20:42:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6600 IOs, 2638104576 bytes 2023-12-08T20:42:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6700 IOs, 2692709376 bytes 2023-12-08T20:42:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6800 IOs, 2778805248 bytes 2023-12-08T20:42:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6900 IOs, 2834140160 bytes 2023-12-08T20:42:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7000 IOs, 2882064384 bytes 2023-12-08T20:42:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7100 IOs, 2911025152 bytes 2023-12-08T20:42:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7200 IOs, 2953210880 bytes 2023-12-08T20:42:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7300 IOs, 3039650816 bytes 2023-12-08T20:42:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7400 IOs, 3109497856 bytes 2023-12-08T20:42:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7500 IOs, 3161207808 bytes 2023-12-08T20:42:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7600 IOs, 3179390976 bytes 2023-12-08T20:42:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7700 IOs, 3275108352 bytes 2023-12-08T20:42:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7800 IOs, 3344404480 bytes 2023-12-08T20:42:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:7900 IOs, 3384233984 bytes 2023-12-08T20:42:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_46 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_46 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_46 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2023-12-08T20:42:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:42:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-08T20:42:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_46 2023-12-08T20:42:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-08T20:42:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:42:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-08T20:42:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_46': 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14733eada03c 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14733eada03c 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:42:45 2023 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:42:45 2023 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:42:45 2023 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:42:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:42:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: ba581ab5-7a74-4534-8eaa-cb18b4444b2d 2023-12-08T20:42:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:42:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:42:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-08T20:42:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_46 2023-12-08T20:42:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:42:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=6 2023-12-08T20:42:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:42:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:42:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 5710848 bytes 2023-12-08T20:42:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 26987520 bytes 2023-12-08T20:42:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 36043776 bytes 2023-12-08T20:42:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 123737088 bytes 2023-12-08T20:42:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 149175296 bytes 2023-12-08T20:42:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 228803584 bytes 2023-12-08T20:42:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 270893056 bytes 2023-12-08T20:42:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 297914368 bytes 2023-12-08T20:42:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 349711360 bytes 2023-12-08T20:42:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 464453632 bytes 2023-12-08T20:42:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 485861376 bytes 2023-12-08T20:42:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 505973760 bytes 2023-12-08T20:42:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 554603520 bytes 2023-12-08T20:42:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 568554496 bytes 2023-12-08T20:42:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 594791424 bytes 2023-12-08T20:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 642995200 bytes 2023-12-08T20:42:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 717047808 bytes 2023-12-08T20:42:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 754307072 bytes 2023-12-08T20:42:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 814710784 bytes 2023-12-08T20:42:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 854214656 bytes 2023-12-08T20:42:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 886965248 bytes 2023-12-08T20:42:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 926380032 bytes 2023-12-08T20:42:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 954353664 bytes 2023-12-08T20:42:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1038833664 bytes 2023-12-08T20:42:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1056919552 bytes 2023-12-08T20:42:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1077665792 bytes 2023-12-08T20:42:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1106017280 bytes 2023-12-08T20:42:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1138562048 bytes 2023-12-08T20:42:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1176651776 bytes 2023-12-08T20:42:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1246120960 bytes 2023-12-08T20:42:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1310577664 bytes 2023-12-08T20:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1354789888 bytes 2023-12-08T20:42:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1392620544 bytes 2023-12-08T20:42:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1393235968 bytes 2023-12-08T20:42:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:42:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:42:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:42:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:42:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_47 2023-12-08T20:42:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_47 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_47 2023-12-08T20:42:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:42:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:42:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:42:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:42:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2023-12-08T20:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-08T20:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_47 2023-12-08T20:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-08T20:42:52.481 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.552772096337044) > weight (0.25) 2023-12-08T20:42:52.481 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3577507381400209) > weight (0.25) 2023-12-08T20:42:52.481 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-08T20:42:52.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-08T20:42:52.482 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-08T20:42:52.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:42:52.472+0000 7fde123c5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 46332) UID: 1000 2023-12-08T20:42:52.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:42:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:42:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-08T20:42:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_47': 2023-12-08T20:42:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 147e6cba6940 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.147e6cba6940 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:42:52 2023 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:42:52 2023 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:42:52 2023 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: a984bc79-0e3a-4142-922a-bd4298675df0 2023-12-08T20:42:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:42:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:42:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-08T20:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_47 2023-12-08T20:42:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:42:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:42:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:42:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:42:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 626688 bytes 2023-12-08T20:42:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 73885696 bytes 2023-12-08T20:42:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 106627072 bytes 2023-12-08T20:42:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 136312832 bytes 2023-12-08T20:42:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 190332928 bytes 2023-12-08T20:42:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 223368192 bytes 2023-12-08T20:42:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 269104128 bytes 2023-12-08T20:42:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 320198656 bytes 2023-12-08T20:42:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 371103744 bytes 2023-12-08T20:42:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 395585536 bytes 2023-12-08T20:42:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 396729344 bytes 2023-12-08T20:42:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 421965824 bytes 2023-12-08T20:42:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 432439296 bytes 2023-12-08T20:42:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 481264640 bytes 2023-12-08T20:42:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 533448704 bytes 2023-12-08T20:42:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 618240000 bytes 2023-12-08T20:42:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 618866688 bytes 2023-12-08T20:42:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 634774528 bytes 2023-12-08T20:42:56.482 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-08T20:42:56.482 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:42:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:42:56.483 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-08T20:42:56.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-08T20:42:56.483 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-08T20:42:56.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-08T20:42:56.486 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-08T20:42:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 635383808 bytes 2023-12-08T20:42:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 670713856 bytes 2023-12-08T20:42:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 686981120 bytes 2023-12-08T20:42:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 734543872 bytes 2023-12-08T20:42:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 830473216 bytes 2023-12-08T20:42:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 858542080 bytes 2023-12-08T20:42:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 859145216 bytes 2023-12-08T20:42:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 877219840 bytes 2023-12-08T20:42:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 917914624 bytes 2023-12-08T20:42:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 954306560 bytes 2023-12-08T20:42:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 958043136 bytes 2023-12-08T20:42:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 988037120 bytes 2023-12-08T20:42:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1004624896 bytes 2023-12-08T20:42:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1037983744 bytes 2023-12-08T20:42:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1062262784 bytes 2023-12-08T20:42:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1135705088 bytes 2023-12-08T20:42:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1176706048 bytes 2023-12-08T20:42:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1219351552 bytes 2023-12-08T20:42:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1286825984 bytes 2023-12-08T20:42:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1320735744 bytes 2023-12-08T20:42:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1385060352 bytes 2023-12-08T20:43:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1481227264 bytes 2023-12-08T20:43:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1558517760 bytes 2023-12-08T20:43:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1584187392 bytes 2023-12-08T20:43:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1591794688 bytes 2023-12-08T20:43:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1651895296 bytes 2023-12-08T20:43:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1710534656 bytes 2023-12-08T20:43:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1761653760 bytes 2023-12-08T20:43:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1800226816 bytes 2023-12-08T20:43:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1848903680 bytes 2023-12-08T20:43:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 1931425792 bytes 2023-12-08T20:43:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2021914624 bytes 2023-12-08T20:43:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2068438016 bytes 2023-12-08T20:43:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_48 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_48 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:43:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_48 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2023-12-08T20:43:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:43:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-08T20:43:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_48 2023-12-08T20:43:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:43:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-08T20:43:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:43:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-08T20:43:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_48': 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 148e4a61c85f 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.148e4a61c85f 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:43:02 2023 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:43:02 2023 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:43:02 2023 2023-12-08T20:43:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:43:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:43:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 87760f99-26be-4679-8b1b-60035bcda1c0 2023-12-08T20:43:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:43:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:43:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-08T20:43:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_48 2023-12-08T20:43:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:43:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=10 2023-12-08T20:43:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:43:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:43:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 76246016 bytes 2023-12-08T20:43:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 100056064 bytes 2023-12-08T20:43:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 119598080 bytes 2023-12-08T20:43:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 132575232 bytes 2023-12-08T20:43:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 149987328 bytes 2023-12-08T20:43:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 180239360 bytes 2023-12-08T20:43:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 193408000 bytes 2023-12-08T20:43:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 223690752 bytes 2023-12-08T20:43:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 250637312 bytes 2023-12-08T20:43:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 319228928 bytes 2023-12-08T20:43:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 346701824 bytes 2023-12-08T20:43:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 369844224 bytes 2023-12-08T20:43:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 381609984 bytes 2023-12-08T20:43:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 382232576 bytes 2023-12-08T20:43:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 415358976 bytes 2023-12-08T20:43:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 478354432 bytes 2023-12-08T20:43:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 509407232 bytes 2023-12-08T20:43:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 534618112 bytes 2023-12-08T20:43:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 556937216 bytes 2023-12-08T20:43:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 609970176 bytes 2023-12-08T20:43:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 680779776 bytes 2023-12-08T20:43:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 686769152 bytes 2023-12-08T20:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 704590848 bytes 2023-12-08T20:43:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 728956928 bytes 2023-12-08T20:43:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 792896512 bytes 2023-12-08T20:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 854089728 bytes 2023-12-08T20:43:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 899313664 bytes 2023-12-08T20:43:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 974610432 bytes 2023-12-08T20:43:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1026167808 bytes 2023-12-08T20:43:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1074088960 bytes 2023-12-08T20:43:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1141661696 bytes 2023-12-08T20:43:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1184358400 bytes 2023-12-08T20:43:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1210619904 bytes 2023-12-08T20:43:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1313371136 bytes 2023-12-08T20:43:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1326804992 bytes 2023-12-08T20:43:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1377175552 bytes 2023-12-08T20:43:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1495110656 bytes 2023-12-08T20:43:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1531399168 bytes 2023-12-08T20:43:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1585468416 bytes 2023-12-08T20:43:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1663136768 bytes 2023-12-08T20:43:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1695685632 bytes 2023-12-08T20:43:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1748323328 bytes 2023-12-08T20:43:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1767361536 bytes 2023-12-08T20:43:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1769142272 bytes 2023-12-08T20:43:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1840783360 bytes 2023-12-08T20:43:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1867806720 bytes 2023-12-08T20:43:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1926545408 bytes 2023-12-08T20:43:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1967241216 bytes 2023-12-08T20:43:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2024122368 bytes 2023-12-08T20:43:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2080043008 bytes 2023-12-08T20:43:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2133071872 bytes 2023-12-08T20:43:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2203437056 bytes 2023-12-08T20:43:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2267122688 bytes 2023-12-08T20:43:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2298078208 bytes 2023-12-08T20:43:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2307113984 bytes 2023-12-08T20:43:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2338675712 bytes 2023-12-08T20:43:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2360165376 bytes 2023-12-08T20:43:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2404505600 bytes 2023-12-08T20:43:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2479396864 bytes 2023-12-08T20:43:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:43:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:43:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:43:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_49 2023-12-08T20:43:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-08T20:43:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_49 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_49 2023-12-08T20:43:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:43:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:43:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:43:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2023-12-08T20:43:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:43:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-08T20:43:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_49 2023-12-08T20:43:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:43:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-08T20:43:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:43:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_49': 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14991c9d545b 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14991c9d545b 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:43:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:43:13 2023 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:43:13 2023 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:43:13 2023 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bebefe76-db0b-4bdc-9824-3f6918a17856 2023-12-08T20:43:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:43:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:43:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-08T20:43:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_49 2023-12-08T20:43:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:43:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=9 2023-12-08T20:43:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:43:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:43:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 41491456 bytes 2023-12-08T20:43:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 141282304 bytes 2023-12-08T20:43:14.837 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:/build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb539603700 time 2023-12-08T20:43:14.826568+0000 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:/build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: 618: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x5589fae13f21] 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: clone() 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:*** Caught signal (Aborted) ** 2023-12-08T20:43:14.838 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: in thread 7fb539603700 thread_name:bstore_aio 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb542799420] 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: gsignal() 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: abort() 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x5589fae13ff0] 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.839 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 8: clone() 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:43:14.819+0000 7fb539603700 -1 bdev(0x558a06ec2400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:43:14.823+0000 7fb539603700 -1 /build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb539603700 time 2023-12-08T20:43:14.826568+0000 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:/build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: 618: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x5589fae13f21] 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: clone() 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:2023-12-08T20:43:14.827+0000 7fb539603700 -1 *** Caught signal (Aborted) ** 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: in thread 7fb539603700 thread_name:bstore_aio 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.840 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb542799420] 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: gsignal() 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: abort() 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x5589fae13ff0] 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 8: clone() 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T20:43:14.841 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.854 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: -112> 2023-12-08T20:43:14.819+0000 7fb539603700 -1 bdev(0x558a06ec2400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-08T20:43:14.854 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: -21> 2023-12-08T20:43:14.823+0000 7fb539603700 -1 /build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb539603700 time 2023-12-08T20:43:14.826568+0000 2023-12-08T20:43:14.854 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:/build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: 618: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-08T20:43:14.854 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.854 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.854 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x5589fae13f21] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: clone() 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 0> 2023-12-08T20:43:14.827+0000 7fb539603700 -1 *** Caught signal (Aborted) ** 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: in thread 7fb539603700 thread_name:bstore_aio 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb542799420] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: gsignal() 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: abort() 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x5589fae13ff0] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.855 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.856 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 8: clone() 2023-12-08T20:43:14.856 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T20:43:14.856 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.868 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: -242> 2023-12-08T20:43:14.819+0000 7fb539603700 -1 bdev(0x558a06ec2400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: -151> 2023-12-08T20:43:14.823+0000 7fb539603700 -1 /build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb539603700 time 2023-12-08T20:43:14.826568+0000 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:/build/ceph-16.2.14-365-g451ceae1/src/blk/kernel/KernelDevice.cc: 618: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x5589fae13f21] 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: clone() 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: -130> 2023-12-08T20:43:14.827+0000 7fb539603700 -1 *** Caught signal (Aborted) ** 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: in thread 7fb539603700 thread_name:bstore_aio 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:14.869 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb542799420] 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2: gsignal() 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 3: abort() 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x5589fae13ff0] 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 5: (KernelDevice::_aio_thread()+0x111f) [0x5589fba628bf] 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589fba6dc71] 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb54278d609] 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 8: clone() 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T20:43:14.870 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr: 2023-12-08T20:43:15.055 INFO:tasks.ceph.cluster2.osd.0.smithi158.stderr:daemon-helper: command failed with exit status 5 2023-12-08T20:43:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 202874880 bytes 2023-12-08T20:43:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 209826816 bytes 2023-12-08T20:43:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 234792960 bytes 2023-12-08T20:43:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 308468736 bytes 2023-12-08T20:43:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 347717632 bytes 2023-12-08T20:43:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 373661696 bytes 2023-12-08T20:43:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 409387008 bytes 2023-12-08T20:43:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 470555648 bytes 2023-12-08T20:43:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 502354944 bytes 2023-12-08T20:43:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 550211584 bytes 2023-12-08T20:43:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 589478912 bytes 2023-12-08T20:43:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 624926720 bytes 2023-12-08T20:43:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 656154624 bytes 2023-12-08T20:43:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 719721472 bytes 2023-12-08T20:43:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 774122496 bytes 2023-12-08T20:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 807632896 bytes 2023-12-08T20:43:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 863893504 bytes 2023-12-08T20:43:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 900246528 bytes 2023-12-08T20:43:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 944721920 bytes 2023-12-08T20:43:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 1032815616 bytes 2023-12-08T20:43:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1120887808 bytes 2023-12-08T20:43:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1147854848 bytes 2023-12-08T20:43:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1257158656 bytes 2023-12-08T20:43:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1280900096 bytes 2023-12-08T20:43:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1323914240 bytes 2023-12-08T20:43:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1372158976 bytes 2023-12-08T20:43:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1398665216 bytes 2023-12-08T20:43:19.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-08T20:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1448727552 bytes 2023-12-08T20:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1471677440 bytes 2023-12-08T20:43:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1472308224 bytes 2023-12-08T20:43:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1550792704 bytes 2023-12-08T20:43:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1581149184 bytes 2023-12-08T20:43:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1673567232 bytes 2023-12-08T20:43:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1728594944 bytes 2023-12-08T20:43:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1801421824 bytes 2023-12-08T20:43:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1850828800 bytes 2023-12-08T20:43:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1891901440 bytes 2023-12-08T20:43:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1969422336 bytes 2023-12-08T20:43:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 2030140416 bytes 2023-12-08T20:43:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 2030757888 bytes 2023-12-08T20:43:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 2068194304 bytes 2023-12-08T20:43:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 2181254144 bytes 2023-12-08T20:43:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 2265140224 bytes 2023-12-08T20:43:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 2287711232 bytes 2023-12-08T20:43:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 2313563136 bytes 2023-12-08T20:43:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 2405412864 bytes 2023-12-08T20:43:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2422557696 bytes 2023-12-08T20:43:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2502879232 bytes 2023-12-08T20:43:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2569868288 bytes 2023-12-08T20:43:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2595531776 bytes 2023-12-08T20:43:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2636972032 bytes 2023-12-08T20:43:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2666667008 bytes 2023-12-08T20:43:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2727502848 bytes 2023-12-08T20:43:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:43:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_50 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_50 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n snapshot ] 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_50 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local size=128 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n 128M ] 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ size=128M 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2023-12-08T20:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ image = image ] 2023-12-08T20:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-08T20:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_50 2023-12-08T20:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local mode=snapshot 2023-12-08T20:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-08T20:43:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Mirroring enabled 2023-12-08T20:43:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-08T20:43:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_50': 2023-12-08T20:43:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:43:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:43:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 1 2023-12-08T20:43:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 14a34b20fdc8 2023-12-08T20:43:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.14a34b20fdc8 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:43:23 2023 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:43:23 2023 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:43:23 2023 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 5110756a-130a-4ef9-9cd1-de6e80c4951b 2023-12-08T20:43:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: true 2023-12-08T20:43:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ -n ] 2023-12-08T20:43:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-08T20:43:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_50 2023-12-08T20:43:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-08T20:43:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local duration=8 2023-12-08T20:43:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set +e 2023-12-08T20:43:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-08T20:43:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:100 IOs, 51499008 bytes 2023-12-08T20:43:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:200 IOs, 88668160 bytes 2023-12-08T20:43:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:300 IOs, 198867968 bytes 2023-12-08T20:43:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:400 IOs, 216756224 bytes 2023-12-08T20:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:500 IOs, 275724288 bytes 2023-12-08T20:43:25.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-08T20:43:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:600 IOs, 363402240 bytes 2023-12-08T20:43:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:700 IOs, 442797056 bytes 2023-12-08T20:43:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:800 IOs, 466819072 bytes 2023-12-08T20:43:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:900 IOs, 484317184 bytes 2023-12-08T20:43:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1000 IOs, 564421632 bytes 2023-12-08T20:43:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1100 IOs, 592357376 bytes 2023-12-08T20:43:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1200 IOs, 660151296 bytes 2023-12-08T20:43:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1300 IOs, 708440064 bytes 2023-12-08T20:43:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1400 IOs, 709053440 bytes 2023-12-08T20:43:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1500 IOs, 722554880 bytes 2023-12-08T20:43:26.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5785484935588202) > weight (0.25) 2023-12-08T20:43:26.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.49390898911073733) > weight (0.25) 2023-12-08T20:43:26.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5141363646590407) > weight (0.25) 2023-12-08T20:43:26.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8771254271015915) > weight (0.25) 2023-12-08T20:43:26.382 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-08T20:43:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1600 IOs, 738231296 bytes 2023-12-08T20:43:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1700 IOs, 801081344 bytes 2023-12-08T20:43:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1800 IOs, 825925632 bytes 2023-12-08T20:43:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:1900 IOs, 895540224 bytes 2023-12-08T20:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2000 IOs, 930875392 bytes 2023-12-08T20:43:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2100 IOs, 996644864 bytes 2023-12-08T20:43:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2200 IOs, 997252096 bytes 2023-12-08T20:43:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2300 IOs, 1005195264 bytes 2023-12-08T20:43:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2400 IOs, 1078027264 bytes 2023-12-08T20:43:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2500 IOs, 1113527296 bytes 2023-12-08T20:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2600 IOs, 1142591488 bytes 2023-12-08T20:43:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2700 IOs, 1167564800 bytes 2023-12-08T20:43:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2800 IOs, 1180781568 bytes 2023-12-08T20:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:2900 IOs, 1246246912 bytes 2023-12-08T20:43:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3000 IOs, 1267555328 bytes 2023-12-08T20:43:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3100 IOs, 1275388928 bytes 2023-12-08T20:43:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3200 IOs, 1309024256 bytes 2023-12-08T20:43:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3300 IOs, 1319010304 bytes 2023-12-08T20:43:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3400 IOs, 1346765824 bytes 2023-12-08T20:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3500 IOs, 1423485952 bytes 2023-12-08T20:43:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3600 IOs, 1445315584 bytes 2023-12-08T20:43:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3700 IOs, 1478043648 bytes 2023-12-08T20:43:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3800 IOs, 1555614720 bytes 2023-12-08T20:43:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:3900 IOs, 1579614208 bytes 2023-12-08T20:43:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4000 IOs, 1660732416 bytes 2023-12-08T20:43:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4100 IOs, 1736213504 bytes 2023-12-08T20:43:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4200 IOs, 1743557632 bytes 2023-12-08T20:43:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4300 IOs, 1750087680 bytes 2023-12-08T20:43:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4400 IOs, 1765954560 bytes 2023-12-08T20:43:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4500 IOs, 1797308416 bytes 2023-12-08T20:43:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4600 IOs, 1878567936 bytes 2023-12-08T20:43:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4700 IOs, 1890147328 bytes 2023-12-08T20:43:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4800 IOs, 1935492096 bytes 2023-12-08T20:43:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:4900 IOs, 2024806400 bytes 2023-12-08T20:43:30.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-08T20:43:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5000 IOs, 2079007744 bytes 2023-12-08T20:43:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5100 IOs, 2149240832 bytes 2023-12-08T20:43:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5200 IOs, 2167945216 bytes 2023-12-08T20:43:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5300 IOs, 2229204992 bytes 2023-12-08T20:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5400 IOs, 2233475072 bytes 2023-12-08T20:43:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5500 IOs, 2307447808 bytes 2023-12-08T20:43:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5600 IOs, 2349167616 bytes 2023-12-08T20:43:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5700 IOs, 2383608832 bytes 2023-12-08T20:43:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5800 IOs, 2428035072 bytes 2023-12-08T20:43:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:5900 IOs, 2469058560 bytes 2023-12-08T20:43:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6000 IOs, 2469684224 bytes 2023-12-08T20:43:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6100 IOs, 2470286336 bytes 2023-12-08T20:43:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6200 IOs, 2477431808 bytes 2023-12-08T20:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6300 IOs, 2496007168 bytes 2023-12-08T20:43:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6400 IOs, 2515183616 bytes 2023-12-08T20:43:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:6500 IOs, 2610975744 bytes 2023-12-08T20:43:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ error_code=124 2023-12-08T20:43:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set -e 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 124 -eq 124 ] 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image_count=50 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local count 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_count=0 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 40 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 1 -ne 1 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-08T20:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + cut -d grep -f 2 images: 2023-12-08T20:43:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:43:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ count=50 2023-12-08T20:43:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 50 = 50 2023-12-08T20:43:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-08T20:43:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state 2023-12-08T20:43:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 40 2023-12-08T20:43:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 1 -ne 1 2023-12-08T20:43:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + grep image health: 2023-12-08T20:43:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd --cluster cluster2 -p mirror mirror pool status 2023-12-08T20:43:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cut -d -f 3 2023-12-08T20:43:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=OK 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test OK = ERROR 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test OK = OK 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image_count=50 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local count 2023-12-08T20:43:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local last_count=0 2023-12-08T20:43:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 40 2023-12-08T20:43:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 1 -ne 1 2023-12-08T20:43:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-08T20:43:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep images: 2023-12-08T20:43:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cut -d -f 2 2023-12-08T20:43:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:43:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ count=50 2023-12-08T20:43:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 50 = 50 2023-12-08T20:43:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-08T20:43:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state 2023-12-08T20:43:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 40 2023-12-08T20:43:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 1 -ne 1 2023-12-08T20:43:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-08T20:43:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep image health: 2023-12-08T20:43:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cut -d -f 3 2023-12-08T20:43:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:rbd: failed to query services: (13) Permission denied 2023-12-08T20:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ state=OK 2023-12-08T20:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test OK = ERROR 2023-12-08T20:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test OK = OK 2023-12-08T20:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ testlog TEST: compare many images 2023-12-08T20:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ tee -a /tmp/tmp.b8SaxlVk4h/rbd-mirror.test.log 2023-12-08T20:43:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ date +%F %T 2023-12-08T20:43:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo 2023-12-08 20:43:32 TEST: compare many images 2023-12-08T20:43:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:2023-12-08 20:43:32 TEST: compare many images 2023-12-08T20:43:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 1 50 2023-12-08T20:43:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_1 2023-12-08T20:43:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-08T20:43:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:43:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:43:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-08T20:43:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:43:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:43:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:43:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:43:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:43:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:43:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:43:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:43:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:43:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:43:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:43:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:43:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:43:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-08T20:43:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:43:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + grep -i Replaying 2023-12-08T20:43:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:echo "state": "Replaying" 2023-12-08T20:43:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:43:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:43:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:43:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-08T20:43:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:43:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_1.status 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_1.status 2023-12-08T20:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:43:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.status 2023-12-08T20:43:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_1.status 2023-12-08T20:43:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_1.status 2023-12-08T20:43:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:43:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-08T20:43:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-08T20:43:35.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-08T20:43:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 119 2023-12-08T20:43:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.status 2023-12-08T20:43:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.status 2023-12-08T20:43:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-08T20:43:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:43:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=119 2023-12-08T20:43:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:43:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-08T20:43:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:43:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 119 = 2023-12-08T20:43:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:43:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-08T20:43:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 119 = 2023-12-08T20:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:43:37.553 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.539+0000 7f670116f700 -1 reset not still connected to 0x55743dd0e340 2023-12-08T20:43:37.562 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.551+0000 7f670116f700 -1 reset not still connected to 0x5574427ab6c0 2023-12-08T20:43:37.568 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.555+0000 7f670116f700 -1 reset not still connected to 0x55744e3f8820 2023-12-08T20:43:37.568 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.555+0000 7f670116f700 -1 reset not still connected to 0x55744e3f9ad0 2023-12-08T20:43:37.572 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x55744e8d0750 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x55744eacfe10 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x55744eef7a00 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x557451f8e000 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x557451f8edd0 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x557454bca5b0 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x557458ae4dd0 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x557458ae5380 2023-12-08T20:43:37.573 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:43:37.563+0000 7f670116f700 -1 reset not still connected to 0x55745b2629c0 2023-12-08T20:43:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-08T20:43:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:43:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:43:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:43:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 119 = 2023-12-08T20:43:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:43:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.status 2023-12-08T20:43:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-08T20:43:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=119 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 119 = 119 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:43:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_1': 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109e89e1a0e7 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109e89e1a0e7 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:43:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:13 2023 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:13 2023 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:13 2023 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 52a3f781-3d39-4a12-bafb-6bc21cbfa607 2023-12-08T20:43:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_1 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.export 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.export 2023-12-08T20:43:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.export 2023-12-08T20:43:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.export 2023-12-08T20:43:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:43:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.export 2023-12-08T20:43:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:43:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.export 2023-12-08T20:43:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_1.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_1.export 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_2 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:43:41.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-08T20:43:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-08T20:43:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-08T20:43:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:43:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:43:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state+ 2023-12-08T20:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:local cluster=cluster1 2023-12-08T20:43:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:43:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:43:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:43:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:43:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:43:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:43:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:43:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:43:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:43:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:43:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:43:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:43:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:43:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:43:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:43:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:43:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:43:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-08T20:43:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:43:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:43:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:43:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-08T20:43:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:43:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_2.status 2023-12-08T20:43:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_2.status 2023-12-08T20:43:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:43:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.status 2023-12-08T20:43:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_2.status 2023-12-08T20:43:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_2.status 2023-12-08T20:43:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:43:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.status 2023-12-08T20:43:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-08T20:43:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-08T20:43:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 121 2023-12-08T20:43:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.status 2023-12-08T20:43:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.status 2023-12-08T20:43:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-08T20:43:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:43:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=121 2023-12-08T20:43:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:43:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.status 2023-12-08T20:43:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.status 2023-12-08T20:43:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-08T20:43:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:43:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:43:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:43:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 121 = 2023-12-08T20:43:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:43:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.status 2023-12-08T20:43:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.status 2023-12-08T20:43:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-08T20:43:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:43:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:43:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:43:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 121 = 2023-12-08T20:43:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:43:46.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-08T20:43:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.status 2023-12-08T20:43:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.status 2023-12-08T20:43:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-08T20:43:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=121 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 121 = 121 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:43:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-08T20:43:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_2': 2023-12-08T20:43:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:43:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:43:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:43:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109e30976b7d 2023-12-08T20:43:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109e30976b7d 2023-12-08T20:43:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:19 2023 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:19 2023 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:19 2023 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 12c34aaf-ffa4-4588-bab2-a4673b2a7cee 2023-12-08T20:43:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_2 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.export 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.export 2023-12-08T20:43:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.export 2023-12-08T20:43:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.export 2023-12-08T20:43:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:43:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.export 2023-12-08T20:43:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:43:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.export 2023-12-08T20:43:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_2.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_2.export 2023-12-08T20:43:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_3 2023-12-08T20:43:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-08T20:43:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:43:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-08T20:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:43:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:43:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:43:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:43:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:43:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:43:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-08T20:43:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:43:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:43:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:43:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:43:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:43:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:43:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-08T20:43:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:43:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_3.status 2023-12-08T20:43:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_3.status 2023-12-08T20:43:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:43:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.status 2023-12-08T20:43:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_3.status 2023-12-08T20:43:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_3.status 2023-12-08T20:43:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:43:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-08T20:43:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-08T20:43:52.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-08T20:43:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 123 2023-12-08T20:43:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.status 2023-12-08T20:43:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.status 2023-12-08T20:43:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-08T20:43:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:43:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=123 2023-12-08T20:43:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:43:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-08T20:43:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=70 2023-12-08T20:43:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 123 = 70 2023-12-08T20:43:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:43:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-08T20:43:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:43:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:43:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:43:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 123 = 2023-12-08T20:43:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:43:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-08T20:43:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:43:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 123 = 2023-12-08T20:43:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:43:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.status 2023-12-08T20:43:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-08T20:43:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:43:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=123 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 123 = 123 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:43:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-08T20:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_3': 2023-12-08T20:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109ea90047a9 2023-12-08T20:43:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109ea90047a9 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:31 2023 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:31 2023 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:31 2023 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:43:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 5d037cda-53a8-49bc-9a00-03b63f0f3612 2023-12-08T20:43:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:43:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-08T20:43:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_3 2023-12-08T20:43:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.export 2023-12-08T20:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.export 2023-12-08T20:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.export 2023-12-08T20:43:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.export 2023-12-08T20:43:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:43:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.export 2023-12-08T20:43:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:43:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.export 2023-12-08T20:43:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_3.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_3.export 2023-12-08T20:43:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:43:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_4 2023-12-08T20:43:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-08T20:43:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:43:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:43:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:43:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:43:57.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-08T20:43:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-08T20:43:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:43:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-08T20:43:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:43:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:43:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:43:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:43:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:43:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-08T20:44:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local+ cluster=cluster1 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:44:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-08T20:44:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-08T20:44:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:44:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:44:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_4.status 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_4.status 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.status 2023-12-08T20:44:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_4.status 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_4.status 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.status 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-08T20:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 125 2023-12-08T20:44:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.status 2023-12-08T20:44:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.status 2023-12-08T20:44:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-08T20:44:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=125 2023-12-08T20:44:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.status 2023-12-08T20:44:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.status 2023-12-08T20:44:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-08T20:44:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 125 = 2023-12-08T20:44:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.status 2023-12-08T20:44:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.status 2023-12-08T20:44:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-08T20:44:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=125 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 125 = 125 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:02.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-08T20:44:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-08T20:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_4': 2023-12-08T20:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 109eaadadd59 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.109eaadadd59 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:37 2023 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:37 2023 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:37 2023 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: caca2379-4196-4a64-ad7b-2144d08a1309 2023-12-08T20:44:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-08T20:44:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_4 2023-12-08T20:44:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.export 2023-12-08T20:44:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.export 2023-12-08T20:44:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.export 2023-12-08T20:44:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.export 2023-12-08T20:44:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.export 2023-12-08T20:44:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.export 2023-12-08T20:44:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_4.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_4.export 2023-12-08T20:44:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_5 2023-12-08T20:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-08T20:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local+ cluster_instance=cluster1 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i+ statelocal 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: cluster=cluster1 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-08T20:44:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:44:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:44:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-08T20:44:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_5.status 2023-12-08T20:44:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_5.status 2023-12-08T20:44:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.status 2023-12-08T20:44:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_5.status 2023-12-08T20:44:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_5.status 2023-12-08T20:44:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-08T20:44:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-08T20:44:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 127 2023-12-08T20:44:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.status 2023-12-08T20:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.status 2023-12-08T20:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-08T20:44:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=127 2023-12-08T20:44:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-08T20:44:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 127 = 2023-12-08T20:44:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-08T20:44:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 127 = 2023-12-08T20:44:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:44:08.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-08T20:44:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-08T20:44:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 127 = 2023-12-08T20:44:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.status 2023-12-08T20:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-08T20:44:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=127 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 127 = 127 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-08T20:44:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_5': 2023-12-08T20:44:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194bb3874f4 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194bb3874f4 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:42 2023 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:42 2023 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:42 2023 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: ad893482-52ac-4c1f-95dc-b30b5b63a8dd 2023-12-08T20:44:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_5 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.export 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.export 2023-12-08T20:44:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.export 2023-12-08T20:44:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.export 2023-12-08T20:44:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.export 2023-12-08T20:44:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.export 2023-12-08T20:44:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_5.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_5.export 2023-12-08T20:44:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_6 2023-12-08T20:44:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-08T20:44:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:13.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-08T20:44:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-08T20:44:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:44:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:44:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:44:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:44:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-08T20:44:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:44:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:44:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-08T20:44:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_6.status 2023-12-08T20:44:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_6.status 2023-12-08T20:44:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.status 2023-12-08T20:44:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_6.status 2023-12-08T20:44:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_6.status 2023-12-08T20:44:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.status 2023-12-08T20:44:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-08T20:44:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-08T20:44:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 129 2023-12-08T20:44:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.status 2023-12-08T20:44:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.status 2023-12-08T20:44:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-08T20:44:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=129 2023-12-08T20:44:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.status 2023-12-08T20:44:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.status 2023-12-08T20:44:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-08T20:44:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 129 = 2023-12-08T20:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.status 2023-12-08T20:44:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.status 2023-12-08T20:44:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-08T20:44:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=129 2023-12-08T20:44:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 129 = 129 2023-12-08T20:44:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-08T20:44:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-08T20:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_6': 2023-12-08T20:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194520db939 2023-12-08T20:44:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194520db939 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:35:54 2023 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:35:54 2023 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:35:54 2023 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: d6992038-914b-4ae5-92f9-4acfb9dca251 2023-12-08T20:44:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_6 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.export 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.export 2023-12-08T20:44:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.export 2023-12-08T20:44:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.export 2023-12-08T20:44:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.export 2023-12-08T20:44:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.export 2023-12-08T20:44:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_6.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_6.export 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_7 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:19.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-08T20:44:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-08T20:44:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-08T20:44:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-08T20:44:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1grep 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: -i state 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:44:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-08T20:44:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:44:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:44:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_7.status 2023-12-08T20:44:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_7.status 2023-12-08T20:44:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.status 2023-12-08T20:44:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_7.status 2023-12-08T20:44:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_7.status 2023-12-08T20:44:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.status 2023-12-08T20:44:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-08T20:44:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-08T20:44:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 131 2023-12-08T20:44:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.status 2023-12-08T20:44:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.status 2023-12-08T20:44:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-08T20:44:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=131 2023-12-08T20:44:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.status 2023-12-08T20:44:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.status 2023-12-08T20:44:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-08T20:44:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 131 = 2023-12-08T20:44:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.status 2023-12-08T20:44:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.status 2023-12-08T20:44:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-08T20:44:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 131 = 2023-12-08T20:44:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:44:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.status 2023-12-08T20:44:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.status 2023-12-08T20:44:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-08T20:44:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=131 2023-12-08T20:44:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 131 = 131 2023-12-08T20:44:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-08T20:44:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:24.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-08T20:44:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-08T20:44:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_7': 2023-12-08T20:44:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 119482d798d5 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.119482d798d5 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:05 2023 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:05 2023 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:05 2023 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 9345084e-0ac5-4beb-bc92-c8db39509c36 2023-12-08T20:44:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_7 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.export 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.export 2023-12-08T20:44:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.export 2023-12-08T20:44:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.export 2023-12-08T20:44:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.export 2023-12-08T20:44:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.export 2023-12-08T20:44:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_7.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_7.export 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_8 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:44:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i state 2023-12-08T20:44:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-08T20:44:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:44:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:44:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-08T20:44:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_8.status 2023-12-08T20:44:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_8.status 2023-12-08T20:44:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.status 2023-12-08T20:44:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_8.status 2023-12-08T20:44:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_8.status 2023-12-08T20:44:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.status 2023-12-08T20:44:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-08T20:44:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-08T20:44:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 133 2023-12-08T20:44:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.status 2023-12-08T20:44:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.status 2023-12-08T20:44:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-08T20:44:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=133 2023-12-08T20:44:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.status 2023-12-08T20:44:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.status 2023-12-08T20:44:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-08T20:44:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 133 = 2023-12-08T20:44:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:29.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-08T20:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.status 2023-12-08T20:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.status 2023-12-08T20:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-08T20:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 133 = 2023-12-08T20:44:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:44:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.status 2023-12-08T20:44:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.status 2023-12-08T20:44:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-08T20:44:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=133 2023-12-08T20:44:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 133 = 133 2023-12-08T20:44:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-08T20:44:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_8': 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194a9feca9c 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194a9feca9c 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:18 2023 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:18 2023 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:18 2023 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: eec2c21a-48a4-4bfd-b1ce-523278c74bed 2023-12-08T20:44:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-08T20:44:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_8 2023-12-08T20:44:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.export 2023-12-08T20:44:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.export 2023-12-08T20:44:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.export 2023-12-08T20:44:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.export 2023-12-08T20:44:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.export 2023-12-08T20:44:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.export 2023-12-08T20:44:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_8.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_8.export 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_9 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-08T20:44:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:35.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-08T20:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-08T20:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-08T20:44:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:44:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:44:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:44:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-08T20:44:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-08T20:44:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:44:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:44:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-08T20:44:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-08T20:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_9.status 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_9.status 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.status 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_9.status 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_9.status 2023-12-08T20:44:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.status 2023-12-08T20:44:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-08T20:44:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-08T20:44:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 135 2023-12-08T20:44:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.status 2023-12-08T20:44:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.status 2023-12-08T20:44:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-08T20:44:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=135 2023-12-08T20:44:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.status 2023-12-08T20:44:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.status 2023-12-08T20:44:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-08T20:44:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 135 = 2023-12-08T20:44:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.status 2023-12-08T20:44:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.status 2023-12-08T20:44:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-08T20:44:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=135 2023-12-08T20:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 135 = 135 2023-12-08T20:44:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-08T20:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_9': 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11942a237ebe 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11942a237ebe 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:33 2023 2023-12-08T20:44:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:33 2023 2023-12-08T20:44:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:33 2023 2023-12-08T20:44:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 68475f85-0f1a-456b-ad04-ebe94e35d0a4 2023-12-08T20:44:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_9 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.export 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.export 2023-12-08T20:44:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.export 2023-12-08T20:44:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.export 2023-12-08T20:44:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.export 2023-12-08T20:44:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.export 2023-12-08T20:44:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_9.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_9.export 2023-12-08T20:44:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_10 2023-12-08T20:44:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-08T20:44:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-08T20:44:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-08T20:44:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:40.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-08T20:44:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-08T20:44:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep+ local -i instance=cluster1 state 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-08T20:44:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + grep -i Replaying 2023-12-08T20:44:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:echo "state": "Replaying" 2023-12-08T20:44:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_10.status 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_10.status 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.status 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_10.status 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo+ cluster1-mirror-image_10.status 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:sed -e s|[/ ]|_|g 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.status 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-08T20:44:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-08T20:44:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 137 2023-12-08T20:44:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.status 2023-12-08T20:44:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.status 2023-12-08T20:44:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-08T20:44:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:42.486 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9948141714929419) > weight (0.25) 2023-12-08T20:44:42.486 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.27070967970572124) > weight (0.25) 2023-12-08T20:44:42.486 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3337692196883252) > weight (0.25) 2023-12-08T20:44:42.486 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-08T20:44:42.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-08T20:44:42.487 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-08T20:44:42.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:44:42.474+0000 7fd8916e0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 38769) UID: 1000 2023-12-08T20:44:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=137 2023-12-08T20:44:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:42.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:44:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.status 2023-12-08T20:44:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.status 2023-12-08T20:44:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-08T20:44:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 137 = 2023-12-08T20:44:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.status 2023-12-08T20:44:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.status 2023-12-08T20:44:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-08T20:44:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 137 = 2023-12-08T20:44:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:44:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.status 2023-12-08T20:44:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.status 2023-12-08T20:44:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-08T20:44:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=137 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 137 = 137 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_10': 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194592b5e89 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194592b5e89 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:36:47 2023 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:36:47 2023 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:36:47 2023 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f5b9b8e7-d3c4-4866-97bf-4f179a9e81f1 2023-12-08T20:44:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-08T20:44:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_10 2023-12-08T20:44:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.export 2023-12-08T20:44:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.export 2023-12-08T20:44:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.export 2023-12-08T20:44:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.export 2023-12-08T20:44:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.export 2023-12-08T20:44:45.488 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-08T20:44:45.488 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:44:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:44:45.488 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-08T20:44:45.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-08T20:44:45.489 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-08T20:44:45.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-08T20:44:45.491 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-08T20:44:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.export 2023-12-08T20:44:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_10.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_10.export 2023-12-08T20:44:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_11 2023-12-08T20:44:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-08T20:44:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:46.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~87s 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local instance=cluster1 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep+ -i state 2023-12-08T20:44:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:sleep 0 2023-12-08T20:44:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-08T20:44:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo+ "state": "Replaying" 2023-12-08T20:44:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i Replaying 2023-12-08T20:44:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_11.status 2023-12-08T20:44:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_11.status 2023-12-08T20:44:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.status 2023-12-08T20:44:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_11.status 2023-12-08T20:44:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_11.status 2023-12-08T20:44:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-08T20:44:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-08T20:44:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 139 2023-12-08T20:44:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.status 2023-12-08T20:44:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.status 2023-12-08T20:44:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-08T20:44:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=139 2023-12-08T20:44:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-08T20:44:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=78 2023-12-08T20:44:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 139 = 78 2023-12-08T20:44:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-08T20:44:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 139 = 2023-12-08T20:44:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:44:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-08T20:44:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 139 = 2023-12-08T20:44:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:44:51.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-08T20:44:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.status 2023-12-08T20:44:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-08T20:44:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=139 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 139 = 139 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-08T20:44:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_11': 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11942de7b5ab 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11942de7b5ab 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:00 2023 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:00 2023 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:00 2023 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 30b31997-991d-4daf-b0e8-576295786fc4 2023-12-08T20:44:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:44:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-08T20:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_11 2023-12-08T20:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.export 2023-12-08T20:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.export 2023-12-08T20:44:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.export 2023-12-08T20:44:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.export 2023-12-08T20:44:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.export 2023-12-08T20:44:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:44:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.export 2023-12-08T20:44:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_11.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_11.export 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_12 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:44:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:44:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:44:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:44:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:44:56.976 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-08T20:44:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-08T20:44:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:44:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:44:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:44:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-08T20:44:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local+ cluster_instance=cluster1 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:44:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:44:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-08T20:44:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:44:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:44:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:44:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:44:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:44:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:44:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:44:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-08T20:44:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:44:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:44:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:44:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:44:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-08T20:44:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:44:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_12.status 2023-12-08T20:44:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_12.status 2023-12-08T20:44:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:44:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.status 2023-12-08T20:44:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_12.status 2023-12-08T20:44:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo+ cluster1-mirror-image_12.status 2023-12-08T20:44:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:sed -e s|[/ ]|_|g 2023-12-08T20:44:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:44:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-08T20:44:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-08T20:44:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 141 2023-12-08T20:44:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.status 2023-12-08T20:44:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:44:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.status 2023-12-08T20:44:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-08T20:44:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:44:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=141 2023-12-08T20:44:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:44:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:44:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:44:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-08T20:44:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=79 2023-12-08T20:44:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 141 = 79 2023-12-08T20:44:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:44:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:44:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:44:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-08T20:44:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 141 = 2023-12-08T20:44:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:44:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:44:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:44:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:44:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:44:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:44:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:44:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-08T20:44:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:44:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:44:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:44:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:44:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:44:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:44:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 141 = 2023-12-08T20:44:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:45:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:45:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:45:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.status 2023-12-08T20:45:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-08T20:45:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=141 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 141 = 141 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:02.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-08T20:45:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_12': 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 119437501fdb 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.119437501fdb 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:07 2023 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:07 2023 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:07 2023 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 3c3d0826-2049-4f88-b8db-1a6dcc0c3b73 2023-12-08T20:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_12 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.export 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.export 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.export 2023-12-08T20:45:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.export 2023-12-08T20:45:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.export 2023-12-08T20:45:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.export 2023-12-08T20:45:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_12.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_12.export 2023-12-08T20:45:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_13 2023-12-08T20:45:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-08T20:45:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-08T20:45:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-08T20:45:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:45:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:45:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ greplocal -i cluster=cluster1 state 2023-12-08T20:45:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:45:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-08T20:45:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:45:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:45:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_13.status 2023-12-08T20:45:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_13.status 2023-12-08T20:45:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.status 2023-12-08T20:45:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_13.status 2023-12-08T20:45:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_13.status 2023-12-08T20:45:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.status 2023-12-08T20:45:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-08T20:45:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-08T20:45:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 143 2023-12-08T20:45:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.status 2023-12-08T20:45:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.status 2023-12-08T20:45:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-08T20:45:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:45:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=143 2023-12-08T20:45:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:45:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.status 2023-12-08T20:45:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.status 2023-12-08T20:45:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-08T20:45:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 143 = 2023-12-08T20:45:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:45:07.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-08T20:45:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.status 2023-12-08T20:45:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.status 2023-12-08T20:45:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-08T20:45:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 143 = 2023-12-08T20:45:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:45:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.status 2023-12-08T20:45:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.status 2023-12-08T20:45:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-08T20:45:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=143 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 143 = 143 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-08T20:45:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_13': 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11948b057f4a 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11948b057f4a 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:15 2023 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:15 2023 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:15 2023 2023-12-08T20:45:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bf4784b3-1a62-4451-b049-ff2a5cbc99eb 2023-12-08T20:45:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_13 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.export 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.export 2023-12-08T20:45:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.export 2023-12-08T20:45:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.export 2023-12-08T20:45:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.export 2023-12-08T20:45:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.export 2023-12-08T20:45:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_13.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_13.export 2023-12-08T20:45:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_14 2023-12-08T20:45:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-08T20:45:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-08T20:45:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:13.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~114s 2023-12-08T20:45:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-08T20:45:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep+ -ilocal state cluster=cluster1 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-08T20:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-08T20:45:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + echo "state": "Replaying" 2023-12-08T20:45:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i Replaying 2023-12-08T20:45:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_14.status 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_14.status 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.status 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_14.status 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_14.status 2023-12-08T20:45:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.status 2023-12-08T20:45:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-08T20:45:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-08T20:45:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 145 2023-12-08T20:45:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.status 2023-12-08T20:45:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.status 2023-12-08T20:45:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-08T20:45:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:45:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=145 2023-12-08T20:45:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:45:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.status 2023-12-08T20:45:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.status 2023-12-08T20:45:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-08T20:45:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 145 = 2023-12-08T20:45:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:45:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.status 2023-12-08T20:45:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.status 2023-12-08T20:45:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-08T20:45:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 145 = 2023-12-08T20:45:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:45:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.status 2023-12-08T20:45:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.status 2023-12-08T20:45:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-08T20:45:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=145 2023-12-08T20:45:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 145 = 145 2023-12-08T20:45:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-08T20:45:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-08T20:45:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_14': 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11941a9ecc3 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11941a9ecc3 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:21 2023 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:21 2023 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:21 2023 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f7b0e482-81c1-456d-bbf0-de74514a24bc 2023-12-08T20:45:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_14 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.export 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.export 2023-12-08T20:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.export 2023-12-08T20:45:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.export 2023-12-08T20:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.export 2023-12-08T20:45:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.export 2023-12-08T20:45:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_14.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_14.export 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_15 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:18.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-08T20:45:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-08T20:45:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-08T20:45:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:45:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:45:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:45:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:45:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-08T20:45:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-08T20:45:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:45:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:45:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-08T20:45:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_15.status 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_15.status 2023-12-08T20:45:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.status 2023-12-08T20:45:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_15.status 2023-12-08T20:45:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_15.status 2023-12-08T20:45:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-08T20:45:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-08T20:45:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 147 2023-12-08T20:45:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.status 2023-12-08T20:45:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.status 2023-12-08T20:45:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-08T20:45:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:45:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=147 2023-12-08T20:45:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:45:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-08T20:45:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=82 2023-12-08T20:45:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 147 = 82 2023-12-08T20:45:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:45:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-08T20:45:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 147 = 2023-12-08T20:45:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:45:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-08T20:45:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 147 = 2023-12-08T20:45:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:45:23.383 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-08T20:45:23.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-08T20:45:23.384 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-08T20:45:23.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:2023-12-08T20:45:23.378+0000 7f5ba835b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 42347) UID: 1000 2023-12-08T20:45:23.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:45:23.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-08T20:45:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.status 2023-12-08T20:45:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-08T20:45:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=147 2023-12-08T20:45:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 147 = 147 2023-12-08T20:45:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-08T20:45:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-08T20:45:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_15': 2023-12-08T20:45:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11944c36db73 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11944c36db73 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:27 2023 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:27 2023 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:27 2023 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: c614153b-e9e9-4295-a51d-042466d7263b 2023-12-08T20:45:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-08T20:45:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_15 2023-12-08T20:45:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.export 2023-12-08T20:45:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.export 2023-12-08T20:45:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.export 2023-12-08T20:45:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.export 2023-12-08T20:45:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.export 2023-12-08T20:45:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.export 2023-12-08T20:45:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_15.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_15.export 2023-12-08T20:45:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_16 2023-12-08T20:45:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-08T20:45:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:27.385 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-08T20:45:27.385 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:45:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:45:27.386 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-08T20:45:27.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-08T20:45:27.386 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-08T20:45:27.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-08T20:45:27.389 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-08T20:45:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-08T20:45:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-08T20:45:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-08T20:45:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-08T20:45:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local+ cluster_instance=cluster1 2023-12-08T20:45:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:45:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:45:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-08T20:45:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-08T20:45:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:45:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:45:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-08T20:45:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_16.status 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_16.status 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.status 2023-12-08T20:45:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_16.status 2023-12-08T20:45:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_16.status 2023-12-08T20:45:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-08T20:45:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-08T20:45:29.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-08T20:45:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 149 2023-12-08T20:45:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.status 2023-12-08T20:45:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.status 2023-12-08T20:45:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-08T20:45:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:45:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=149 2023-12-08T20:45:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:45:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-08T20:45:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=83 2023-12-08T20:45:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 149 = 83 2023-12-08T20:45:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:45:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-08T20:45:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 149 = 2023-12-08T20:45:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:45:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-08T20:45:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 149 = 2023-12-08T20:45:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.status 2023-12-08T20:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-08T20:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=149 2023-12-08T20:45:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 149 = 149 2023-12-08T20:45:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-08T20:45:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-08T20:45:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_16': 2023-12-08T20:45:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194c5cb4786 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194c5cb4786 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:41 2023 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:41 2023 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:41 2023 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: b5dd2e11-6d62-42d1-b7e9-d63f11c0fc3c 2023-12-08T20:45:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-08T20:45:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_16 2023-12-08T20:45:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.export 2023-12-08T20:45:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.export 2023-12-08T20:45:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.export 2023-12-08T20:45:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.export 2023-12-08T20:45:34.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-08T20:45:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.export 2023-12-08T20:45:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.export 2023-12-08T20:45:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_16.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_16.export 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_17 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-08T20:45:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-08T20:45:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:45:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:45:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state+ 2023-12-08T20:45:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:local instance=cluster1 2023-12-08T20:45:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-08T20:45:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:45:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:45:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_17.status 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + echo cluster2-mirror-image_17.status 2023-12-08T20:45:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:sed -e s|[/ ]|_|g 2023-12-08T20:45:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.status 2023-12-08T20:45:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_17.status 2023-12-08T20:45:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_17.status 2023-12-08T20:45:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.status 2023-12-08T20:45:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-08T20:45:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-08T20:45:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 151 2023-12-08T20:45:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.status 2023-12-08T20:45:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.status 2023-12-08T20:45:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-08T20:45:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:45:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=151 2023-12-08T20:45:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:45:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.status 2023-12-08T20:45:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.status 2023-12-08T20:45:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-08T20:45:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 151 = 2023-12-08T20:45:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:45:40.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2023-12-08T20:45:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.status 2023-12-08T20:45:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.status 2023-12-08T20:45:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-08T20:45:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 151 = 2023-12-08T20:45:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.status 2023-12-08T20:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.status 2023-12-08T20:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-08T20:45:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=151 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 151 = 151 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-08T20:45:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_17': 2023-12-08T20:45:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 119480d59f71 2023-12-08T20:45:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.119480d59f71 2023-12-08T20:45:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:47 2023 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:47 2023 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:47 2023 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: d53aaa49-211e-4a25-8cfe-c474c72c1761 2023-12-08T20:45:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-08T20:45:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_17 2023-12-08T20:45:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.export 2023-12-08T20:45:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.export 2023-12-08T20:45:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.export 2023-12-08T20:45:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.export 2023-12-08T20:45:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.export 2023-12-08T20:45:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.export 2023-12-08T20:45:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_17.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_17.export 2023-12-08T20:45:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_18 2023-12-08T20:45:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-08T20:45:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-08T20:45:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shiftgrep 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: -i state 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-08T20:45:45.610 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-08T20:45:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + grep -i Replaying 2023-12-08T20:45:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:echo "state": "Replaying" 2023-12-08T20:45:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_18.status 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_18.status 2023-12-08T20:45:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.status 2023-12-08T20:45:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_18.status 2023-12-08T20:45:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_18.status 2023-12-08T20:45:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.status 2023-12-08T20:45:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-08T20:45:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-08T20:45:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 153 2023-12-08T20:45:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.status 2023-12-08T20:45:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.status 2023-12-08T20:45:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-08T20:45:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:45:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=153 2023-12-08T20:45:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:45:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.status 2023-12-08T20:45:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.status 2023-12-08T20:45:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-08T20:45:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 153 = 2023-12-08T20:45:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:45:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.status 2023-12-08T20:45:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.status 2023-12-08T20:45:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-08T20:45:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 153 = 2023-12-08T20:45:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:45:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.status 2023-12-08T20:45:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.status 2023-12-08T20:45:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-08T20:45:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=153 2023-12-08T20:45:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 153 = 153 2023-12-08T20:45:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-08T20:45:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_18': 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11945c412a70 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11945c412a70 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:37:54 2023 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:37:54 2023 2023-12-08T20:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:37:54 2023 2023-12-08T20:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bf95c8e1-6877-4faf-91e1-cabad7898b86 2023-12-08T20:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-08T20:45:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_18 2023-12-08T20:45:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.export 2023-12-08T20:45:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.export 2023-12-08T20:45:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.export 2023-12-08T20:45:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.export 2023-12-08T20:45:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.export 2023-12-08T20:45:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.export 2023-12-08T20:45:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_18.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_18.export 2023-12-08T20:45:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_19 2023-12-08T20:45:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-08T20:45:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:51.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-08T20:45:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-08T20:45:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-08T20:45:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-08T20:45:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster=cluster1 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state+ 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:local instance=cluster1 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-08T20:45:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-08T20:45:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + grep -i Replaying 2023-12-08T20:45:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:echo "state": "Replaying" 2023-12-08T20:45:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-08T20:45:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_19.status 2023-12-08T20:45:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_19.status 2023-12-08T20:45:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.status 2023-12-08T20:45:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_19.status 2023-12-08T20:45:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_19.status 2023-12-08T20:45:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.status 2023-12-08T20:45:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-08T20:45:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-08T20:45:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 155 2023-12-08T20:45:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.status 2023-12-08T20:45:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.status 2023-12-08T20:45:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-08T20:45:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:45:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=155 2023-12-08T20:45:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:45:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.status 2023-12-08T20:45:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.status 2023-12-08T20:45:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-08T20:45:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 155 = 2023-12-08T20:45:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.status 2023-12-08T20:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.status 2023-12-08T20:45:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-08T20:45:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:45:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:45:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:45:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:45:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:45:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 155 = 2023-12-08T20:45:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:45:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.status 2023-12-08T20:45:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.status 2023-12-08T20:45:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-08T20:45:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:45:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:45:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=155 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 155 = 155 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-08T20:45:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_19': 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11948343737f 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11948343737f 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:09 2023 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:09 2023 2023-12-08T20:45:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:09 2023 2023-12-08T20:45:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:45:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:45:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:45:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 04e61c39-ed38-426f-95be-662b7ae37286 2023-12-08T20:45:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:45:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_19 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.export 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.export 2023-12-08T20:45:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.export 2023-12-08T20:45:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.export 2023-12-08T20:45:56.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-08T20:45:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.export 2023-12-08T20:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:45:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.export 2023-12-08T20:45:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_19.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_19.export 2023-12-08T20:45:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_20 2023-12-08T20:45:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-08T20:45:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:45:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:45:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:45:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-08T20:45:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:45:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-08T20:45:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:45:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:45:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:45:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:45:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-08T20:45:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:45:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:45:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:45:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:45:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:45:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:45:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:45:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:45:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:45:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:45:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:45:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:45:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-08T20:45:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:45:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:45:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:45:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-08T20:45:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:45:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:45:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_20.status 2023-12-08T20:45:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_20.status 2023-12-08T20:45:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.status 2023-12-08T20:45:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_20.status 2023-12-08T20:45:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_20.status 2023-12-08T20:45:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:45:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.status 2023-12-08T20:45:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-08T20:45:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:45:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:45:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:45:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-08T20:46:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 157 2023-12-08T20:46:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.status 2023-12-08T20:46:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:46:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.status 2023-12-08T20:46:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-08T20:46:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=157 2023-12-08T20:46:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.status 2023-12-08T20:46:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:46:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.status 2023-12-08T20:46:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-08T20:46:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 157 = 2023-12-08T20:46:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.status 2023-12-08T20:46:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:46:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.status 2023-12-08T20:46:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-08T20:46:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:01.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 157 = 2023-12-08T20:46:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:01.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-08T20:46:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.status 2023-12-08T20:46:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:46:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.status 2023-12-08T20:46:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-08T20:46:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=157 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 157 = 157 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_20': 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11941ac61569 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11941ac61569 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:21 2023 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:21 2023 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:21 2023 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 9efe6779-be3c-472b-840a-27713ee6053d 2023-12-08T20:46:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_20 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.export 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.export 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.export 2023-12-08T20:46:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.export 2023-12-08T20:46:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.export 2023-12-08T20:46:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.export 2023-12-08T20:46:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_20.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_20.export 2023-12-08T20:46:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_21 2023-12-08T20:46:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-08T20:46:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-08T20:46:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + grepshift -i 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:46:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-08T20:46:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo+ "state": "Replaying" 2023-12-08T20:46:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i Replaying 2023-12-08T20:46:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-08T20:46:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_21.status 2023-12-08T20:46:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_21.status 2023-12-08T20:46:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.status 2023-12-08T20:46:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_21.status 2023-12-08T20:46:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_21.status 2023-12-08T20:46:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.status 2023-12-08T20:46:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-08T20:46:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-08T20:46:07.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-08T20:46:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 159 2023-12-08T20:46:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.status 2023-12-08T20:46:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.status 2023-12-08T20:46:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-08T20:46:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=159 2023-12-08T20:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.status 2023-12-08T20:46:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.status 2023-12-08T20:46:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-08T20:46:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 159 = 2023-12-08T20:46:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:08.308 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.290+0000 7f2c70979700 -1 reset not still connected to 0x55967e16b520 2023-12-08T20:46:08.308 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.290+0000 7f2c70979700 -1 reset not still connected to 0x55967e16b6c0 2023-12-08T20:46:08.308 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.290+0000 7f2c70979700 -1 reset not still connected to 0x55967e315040 2023-12-08T20:46:08.308 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.290+0000 7f2c70979700 -1 reset not still connected to 0x55967e315ee0 2023-12-08T20:46:08.309 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55967efde000 2023-12-08T20:46:08.310 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55967fa80680 2023-12-08T20:46:08.310 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55967fa80d00 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x559683d5d1e0 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x559683d5d6c0 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x559683d5da00 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x5596842b52b0 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55968f064ea0 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55968f0655f0 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55968f594dd0 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55968f5952b0 2023-12-08T20:46:08.311 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T20:46:08.294+0000 7f2c70979700 -1 reset not still connected to 0x55968f595930 2023-12-08T20:46:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.status 2023-12-08T20:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.status 2023-12-08T20:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-08T20:46:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 159 = 2023-12-08T20:46:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.status 2023-12-08T20:46:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.status 2023-12-08T20:46:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-08T20:46:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=159 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 159 = 159 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-08T20:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_21': 2023-12-08T20:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 119444ecf0aa 2023-12-08T20:46:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.119444ecf0aa 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:27 2023 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:27 2023 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:27 2023 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f344c1db-e806-48ca-a049-9d5faa56ca26 2023-12-08T20:46:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-08T20:46:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_21 2023-12-08T20:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.export 2023-12-08T20:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.export 2023-12-08T20:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.export 2023-12-08T20:46:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.export 2023-12-08T20:46:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.export 2023-12-08T20:46:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.export 2023-12-08T20:46:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_21.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_21.export 2023-12-08T20:46:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_22 2023-12-08T20:46:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-08T20:46:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:12.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local+ loop_instance 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grepsleep -i 0 state 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-08T20:46:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying"+ 2023-12-08T20:46:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i Replaying 2023-12-08T20:46:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-08T20:46:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_22.status 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_22.status 2023-12-08T20:46:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.status 2023-12-08T20:46:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_22.status 2023-12-08T20:46:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_22.status 2023-12-08T20:46:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.status 2023-12-08T20:46:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-08T20:46:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-08T20:46:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 161 2023-12-08T20:46:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.status 2023-12-08T20:46:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.status 2023-12-08T20:46:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-08T20:46:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=161 2023-12-08T20:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.status 2023-12-08T20:46:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.status 2023-12-08T20:46:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-08T20:46:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 161 = 2023-12-08T20:46:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.status 2023-12-08T20:46:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.status 2023-12-08T20:46:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-08T20:46:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 161 = 2023-12-08T20:46:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.status 2023-12-08T20:46:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.status 2023-12-08T20:46:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-08T20:46:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=161 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 161 = 161 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_22': 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 119487adb651 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.119487adb651 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:37 2023 2023-12-08T20:46:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:37 2023 2023-12-08T20:46:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:37 2023 2023-12-08T20:46:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 23fc213f-3b3f-4a6e-8700-6338e36a234a 2023-12-08T20:46:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_22 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.export 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.export 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.export 2023-12-08T20:46:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.export 2023-12-08T20:46:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.export 2023-12-08T20:46:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.export 2023-12-08T20:46:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_22.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_22.export 2023-12-08T20:46:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_23 2023-12-08T20:46:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-08T20:46:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:18.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-08T20:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-08T20:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-08T20:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-08T20:46:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:46:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i state 2023-12-08T20:46:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-08T20:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-08T20:46:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:46:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:46:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-08T20:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_23.status 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_23.status 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.status 2023-12-08T20:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_23.status 2023-12-08T20:46:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_23.status 2023-12-08T20:46:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-08T20:46:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-08T20:46:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 163 2023-12-08T20:46:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.status 2023-12-08T20:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.status 2023-12-08T20:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-08T20:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=163 2023-12-08T20:46:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-08T20:46:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=90 2023-12-08T20:46:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 163 = 90 2023-12-08T20:46:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-08T20:46:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 163 = 2023-12-08T20:46:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-08T20:46:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:22.914 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.905+0000 7f670116f700 -1 reset not still connected to 0x55743daaeb60 2023-12-08T20:46:22.914 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.905+0000 7f670116f700 -1 reset not still connected to 0x5574427aa410 2023-12-08T20:46:22.916 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.905+0000 7f670116f700 -1 reset not still connected to 0x5574427ab6c0 2023-12-08T20:46:22.916 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.905+0000 7f670116f700 -1 reset not still connected to 0x5574427aba00 2023-12-08T20:46:22.916 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.905+0000 7f670116f700 -1 reset not still connected to 0x55744e3f8820 2023-12-08T20:46:22.921 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x55744e8d0750 2023-12-08T20:46:22.921 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x55744eacfe10 2023-12-08T20:46:22.921 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x55744eef7a00 2023-12-08T20:46:22.921 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x557451f8edd0 2023-12-08T20:46:22.921 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x557454bca5b0 2023-12-08T20:46:22.923 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x557458ae4dd0 2023-12-08T20:46:22.923 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x557458ae5380 2023-12-08T20:46:22.923 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:46:22.913+0000 7f670116f700 -1 reset not still connected to 0x55745b2629c0 2023-12-08T20:46:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 163 = 2023-12-08T20:46:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:46:23.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-08T20:46:24.492 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8613311386424112) > weight (0.25) 2023-12-08T20:46:24.492 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7814461907747269) > weight (0.25) 2023-12-08T20:46:24.492 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-08T20:46:24.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-08T20:46:24.493 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-08T20:46:24.493 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-08T20:46:24.493 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:46:24.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:46:24.485+0000 7f48135a4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 47307) UID: 1000 2023-12-08T20:46:24.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.status 2023-12-08T20:46:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-08T20:46:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=163 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 163 = 163 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:24.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:46:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_23': 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11943a6e8d06 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11943a6e8d06 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:38:52 2023 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:38:52 2023 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:38:52 2023 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: d02d2b15-17e5-49e1-a2bb-193d056e8988 2023-12-08T20:46:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_23 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.export 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.export 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.export 2023-12-08T20:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.export 2023-12-08T20:46:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.export 2023-12-08T20:46:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.export 2023-12-08T20:46:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_23.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_23.export 2023-12-08T20:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_24 2023-12-08T20:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-08T20:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:28.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster=cluster1 2023-12-08T20:46:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:46:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ greplocal loop_instance 2023-12-08T20:46:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: -i state 2023-12-08T20:46:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:46:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-08T20:46:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-08T20:46:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:46:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:46:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-08T20:46:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_24.status 2023-12-08T20:46:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_24.status 2023-12-08T20:46:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.status 2023-12-08T20:46:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_24.status 2023-12-08T20:46:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_24.status 2023-12-08T20:46:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-08T20:46:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-08T20:46:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 165 2023-12-08T20:46:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.status 2023-12-08T20:46:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.status 2023-12-08T20:46:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-08T20:46:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=165 2023-12-08T20:46:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-08T20:46:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=91 2023-12-08T20:46:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 165 = 91 2023-12-08T20:46:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:46:30.595 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-08T20:46:30.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-08T20:46:30.596 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-08T20:46:30.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-08T20:46:30.598 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-08T20:46:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-08T20:46:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 165 = 2023-12-08T20:46:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-08T20:46:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 165 = 2023-12-08T20:46:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:46:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.status 2023-12-08T20:46:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-08T20:46:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=165 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 165 = 165 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:34.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-08T20:46:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_24': 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194946e6ead 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194946e6ead 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:00 2023 2023-12-08T20:46:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:00 2023 2023-12-08T20:46:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:00 2023 2023-12-08T20:46:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0417c7e3-5aea-47f6-9a2e-317d60062a37 2023-12-08T20:46:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_24 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.export 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.export 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.export 2023-12-08T20:46:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.export 2023-12-08T20:46:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.export 2023-12-08T20:46:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.export 2023-12-08T20:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_24.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_24.export 2023-12-08T20:46:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_25 2023-12-08T20:46:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-08T20:46:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:37.390 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-08T20:46:37.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-08T20:46:37.390 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-08T20:46:37.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:2023-12-08T20:46:37.381+0000 7f2d9e9cd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51118) UID: 1000 2023-12-08T20:46:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-08T20:46:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:37.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-08T20:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local+ cluster=cluster1 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ greplocal -i loop_instance 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state 2023-12-08T20:46:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:46:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-08T20:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-08T20:46:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:46:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:46:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_25.status 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_25.status 2023-12-08T20:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.status 2023-12-08T20:46:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_25.status 2023-12-08T20:46:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_25.status 2023-12-08T20:46:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.status 2023-12-08T20:46:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-08T20:46:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-08T20:46:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 167 2023-12-08T20:46:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.status 2023-12-08T20:46:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.status 2023-12-08T20:46:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-08T20:46:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=167 2023-12-08T20:46:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:39.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-08T20:46:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.status 2023-12-08T20:46:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.status 2023-12-08T20:46:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-08T20:46:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 167 = 2023-12-08T20:46:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.status 2023-12-08T20:46:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.status 2023-12-08T20:46:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-08T20:46:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 167 = 2023-12-08T20:46:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.status 2023-12-08T20:46:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.status 2023-12-08T20:46:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-08T20:46:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=167 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 167 = 167 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-08T20:46:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_25': 2023-12-08T20:46:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194d82705ec 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194d82705ec 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:06 2023 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:06 2023 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:06 2023 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 0cc3114a-15b3-4921-9172-57219de8ca5a 2023-12-08T20:46:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-08T20:46:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_25 2023-12-08T20:46:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.export 2023-12-08T20:46:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.export 2023-12-08T20:46:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.export 2023-12-08T20:46:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.export 2023-12-08T20:46:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.export 2023-12-08T20:46:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.export 2023-12-08T20:46:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_25.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_25.export 2023-12-08T20:46:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_26 2023-12-08T20:46:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-08T20:46:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:45.053 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-08T20:46:45.391 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-08T20:46:45.392 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:46:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:46:45.392 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-08T20:46:45.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-08T20:46:45.392 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-08T20:46:45.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-08T20:46:45.395 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-12-08T20:46:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-08T20:46:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster=cluster1 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep+ -ilocal state instance=cluster1 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:46:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-08T20:46:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:46:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:46:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_26.status 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_26.status 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.status 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_26.status 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_26.status 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.status 2023-12-08T20:46:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-08T20:46:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-08T20:46:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 169 2023-12-08T20:46:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.status 2023-12-08T20:46:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.status 2023-12-08T20:46:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-08T20:46:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=169 2023-12-08T20:46:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.status 2023-12-08T20:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.status 2023-12-08T20:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-08T20:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 169 = 2023-12-08T20:46:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.status 2023-12-08T20:46:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.status 2023-12-08T20:46:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-08T20:46:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 169 = 2023-12-08T20:46:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:47.868 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:46:47.857+0000 7f587a51c700 -1 reset not still connected to 0x55bc44b841a0 2023-12-08T20:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.status 2023-12-08T20:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.status 2023-12-08T20:46:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-08T20:46:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=169 2023-12-08T20:46:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 169 = 169 2023-12-08T20:46:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-08T20:46:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_26': 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11949e204945 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11949e204945 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:17 2023 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:17 2023 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:17 2023 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 83bb9742-972b-4f04-9ef2-ea6466b8f121 2023-12-08T20:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_26 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.export 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.export 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.export 2023-12-08T20:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.export 2023-12-08T20:46:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.export 2023-12-08T20:46:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.export 2023-12-08T20:46:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_26.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_26.export 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_27 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:50.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-08T20:46:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-08T20:46:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-08T20:46:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-08T20:46:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster=cluster1 2023-12-08T20:46:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:46:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep+ -ilocal loop_instance state 2023-12-08T20:46:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:46:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-08T20:46:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-08T20:46:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:46:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:46:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-08T20:46:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_27.status 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_27.status 2023-12-08T20:46:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.status 2023-12-08T20:46:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_27.status 2023-12-08T20:46:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_27.status 2023-12-08T20:46:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.status 2023-12-08T20:46:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-08T20:46:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-08T20:46:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 171 2023-12-08T20:46:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.status 2023-12-08T20:46:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.status 2023-12-08T20:46:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-08T20:46:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:46:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=171 2023-12-08T20:46:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:46:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.status 2023-12-08T20:46:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.status 2023-12-08T20:46:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-08T20:46:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 171 = 2023-12-08T20:46:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:46:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.status 2023-12-08T20:46:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.status 2023-12-08T20:46:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-08T20:46:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:46:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:46:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:46:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:46:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:46:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 171 = 2023-12-08T20:46:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:46:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.status 2023-12-08T20:46:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.status 2023-12-08T20:46:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-08T20:46:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:46:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:46:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=171 2023-12-08T20:46:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 171 = 171 2023-12-08T20:46:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-08T20:46:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:55.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-08T20:46:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_27': 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194ce719451 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194ce719451 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:46:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:27 2023 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:27 2023 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:27 2023 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 47bc65ba-c977-4038-853c-dcaeac323616 2023-12-08T20:46:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_27 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.export 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.export 2023-12-08T20:46:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.export 2023-12-08T20:46:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.export 2023-12-08T20:46:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.export 2023-12-08T20:46:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:46:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.export 2023-12-08T20:46:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_27.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_27.export 2023-12-08T20:46:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_28 2023-12-08T20:46:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-08T20:46:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:46:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:46:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:46:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:46:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:46:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:46:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-08T20:46:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:46:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:46:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:46:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:46:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-08T20:46:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:46:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:46:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:46:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:46:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:46:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:46:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:46:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:46:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-08T20:46:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:46:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:46:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:46:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:46:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:46:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-08T20:46:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:46:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:46:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:46:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_28.status 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_28.status 2023-12-08T20:46:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.status 2023-12-08T20:46:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_28.status 2023-12-08T20:46:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_28.status 2023-12-08T20:46:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:46:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.status 2023-12-08T20:46:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-08T20:46:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:46:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:46:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:46:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-08T20:47:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 173 2023-12-08T20:47:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.status 2023-12-08T20:47:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:47:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.status 2023-12-08T20:47:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-08T20:47:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=173 2023-12-08T20:47:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.status 2023-12-08T20:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:47:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.status 2023-12-08T20:47:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-08T20:47:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 173 = 2023-12-08T20:47:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:01.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-12-08T20:47:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.status 2023-12-08T20:47:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:47:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.status 2023-12-08T20:47:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-08T20:47:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 173 = 2023-12-08T20:47:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.status 2023-12-08T20:47:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:47:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.status 2023-12-08T20:47:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-08T20:47:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=173 2023-12-08T20:47:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 173 = 173 2023-12-08T20:47:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-08T20:47:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:47:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_28': 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 119458e36f2e 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.119458e36f2e 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:34 2023 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:34 2023 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:34 2023 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: da955d28-5776-48a9-8c45-194a8e053c29 2023-12-08T20:47:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-08T20:47:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_28 2023-12-08T20:47:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.export 2023-12-08T20:47:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.export 2023-12-08T20:47:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.export 2023-12-08T20:47:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.export 2023-12-08T20:47:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.export 2023-12-08T20:47:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.export 2023-12-08T20:47:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_28.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_28.export 2023-12-08T20:47:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_29 2023-12-08T20:47:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-08T20:47:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:47:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-08T20:47:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:47:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:47:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:47:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-08T20:47:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:47:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local instance=cluster1 2023-12-08T20:47:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:47:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state+ 2023-12-08T20:47:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:sleep 0 2023-12-08T20:47:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:47:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:47:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-08T20:47:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:47:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:47:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:47:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:47:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:47:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:47:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:47:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:47:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:47:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:47:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:47:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-08T20:47:06.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-08T20:47:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:47:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:47:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:47:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:47:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_29.status 2023-12-08T20:47:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_29.status 2023-12-08T20:47:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.status 2023-12-08T20:47:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_29.status 2023-12-08T20:47:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_29.status 2023-12-08T20:47:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.status 2023-12-08T20:47:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-08T20:47:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-08T20:47:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 175 2023-12-08T20:47:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.status 2023-12-08T20:47:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.status 2023-12-08T20:47:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-08T20:47:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=175 2023-12-08T20:47:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.status 2023-12-08T20:47:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.status 2023-12-08T20:47:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-08T20:47:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 175 = 2023-12-08T20:47:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.status 2023-12-08T20:47:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.status 2023-12-08T20:47:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-08T20:47:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 175 = 2023-12-08T20:47:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.status 2023-12-08T20:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.status 2023-12-08T20:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-08T20:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=175 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 175 = 175 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-08T20:47:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_29': 2023-12-08T20:47:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 119428ec114f 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.119428ec114f 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:43 2023 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:43 2023 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:43 2023 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 9f548ada-9bca-4758-8a9d-577afa105e40 2023-12-08T20:47:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-08T20:47:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_29 2023-12-08T20:47:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.export 2023-12-08T20:47:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.export 2023-12-08T20:47:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.export 2023-12-08T20:47:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.export 2023-12-08T20:47:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.export 2023-12-08T20:47:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.export 2023-12-08T20:47:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_29.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_29.export 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_30 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:47:12.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-08T20:47:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-08T20:47:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:47:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:47:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:47:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-08T20:47:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_30.status 2023-12-08T20:47:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_30.status 2023-12-08T20:47:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.status 2023-12-08T20:47:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_30.status 2023-12-08T20:47:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_30.status 2023-12-08T20:47:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.status 2023-12-08T20:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-08T20:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-08T20:47:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 177 2023-12-08T20:47:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.status 2023-12-08T20:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.status 2023-12-08T20:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-08T20:47:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=177 2023-12-08T20:47:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.status 2023-12-08T20:47:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.status 2023-12-08T20:47:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-08T20:47:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 177 = 2023-12-08T20:47:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.status 2023-12-08T20:47:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.status 2023-12-08T20:47:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-08T20:47:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 177 = 2023-12-08T20:47:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.status 2023-12-08T20:47:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.status 2023-12-08T20:47:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-08T20:47:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=177 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 177 = 177 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:17.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-12-08T20:47:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-08T20:47:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_30': 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194a82c8d91 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194a82c8d91 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:39:53 2023 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:39:53 2023 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:39:53 2023 2023-12-08T20:47:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 2d94a5cb-6bdc-46af-9117-bdba7256b886 2023-12-08T20:47:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_30 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.export 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.export 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.export 2023-12-08T20:47:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.export 2023-12-08T20:47:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.export 2023-12-08T20:47:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.export 2023-12-08T20:47:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_30.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_30.export 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_31 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:47:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-08T20:47:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-08T20:47:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:47:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-08T20:47:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i+ state 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:local cluster=cluster1 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:47:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:47:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:47:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:47:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:47:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-08T20:47:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:47:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:47:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:47:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:47:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_31.status 2023-12-08T20:47:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_31.status 2023-12-08T20:47:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.status 2023-12-08T20:47:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_31.status 2023-12-08T20:47:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + echo cluster1-mirror-image_31.status 2023-12-08T20:47:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:sed -e s|[/ ]|_|g 2023-12-08T20:47:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-08T20:47:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-08T20:47:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 179 2023-12-08T20:47:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.status 2023-12-08T20:47:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.status 2023-12-08T20:47:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-08T20:47:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=179 2023-12-08T20:47:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-08T20:47:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=98 2023-12-08T20:47:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 179 = 98 2023-12-08T20:47:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:22.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-08T20:47:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-08T20:47:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 179 = 2023-12-08T20:47:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-08T20:47:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 179 = 2023-12-08T20:47:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.status 2023-12-08T20:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-08T20:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=179 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 179 = 179 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-08T20:47:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_31': 2023-12-08T20:47:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194220fa282 2023-12-08T20:47:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194220fa282 2023-12-08T20:47:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:01 2023 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:01 2023 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:01 2023 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 4250ab73-a265-4e1d-b62b-01a21da69296 2023-12-08T20:47:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-08T20:47:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_31 2023-12-08T20:47:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.export 2023-12-08T20:47:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.export 2023-12-08T20:47:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.export 2023-12-08T20:47:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.export 2023-12-08T20:47:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.export 2023-12-08T20:47:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.export 2023-12-08T20:47:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_31.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_31.export 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_32 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:47:28.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-12-08T20:47:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-08T20:47:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-08T20:47:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1+ 2023-12-08T20:47:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grepshift -i 2023-12-08T20:47:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: state 2023-12-08T20:47:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:47:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:47:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:47:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:47:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:47:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-08T20:47:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:47:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:47:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:47:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:47:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-08T20:47:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:47:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:47:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:47:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:47:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:47:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:47:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-08T20:47:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_32.status 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_32.status 2023-12-08T20:47:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.status 2023-12-08T20:47:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_32.status 2023-12-08T20:47:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_32.status 2023-12-08T20:47:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.status 2023-12-08T20:47:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-08T20:47:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-08T20:47:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 181 2023-12-08T20:47:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.status 2023-12-08T20:47:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.status 2023-12-08T20:47:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-08T20:47:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=181 2023-12-08T20:47:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.status 2023-12-08T20:47:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.status 2023-12-08T20:47:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-08T20:47:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 181 = 2023-12-08T20:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.status 2023-12-08T20:47:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.status 2023-12-08T20:47:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-08T20:47:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 181 = 2023-12-08T20:47:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.status 2023-12-08T20:47:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.status 2023-12-08T20:47:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-08T20:47:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=181 2023-12-08T20:47:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 181 = 181 2023-12-08T20:47:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-08T20:47:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-08T20:47:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_32': 2023-12-08T20:47:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194c9a618c3 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194c9a618c3 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:12 2023 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:12 2023 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:12 2023 2023-12-08T20:47:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: f540800b-9391-4c36-bc38-ff53eed6a35c 2023-12-08T20:47:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-08T20:47:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_32 2023-12-08T20:47:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.export 2023-12-08T20:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.export 2023-12-08T20:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.export 2023-12-08T20:47:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.export 2023-12-08T20:47:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.export 2023-12-08T20:47:33.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-12-08T20:47:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.export 2023-12-08T20:47:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_32.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_32.export 2023-12-08T20:47:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_33 2023-12-08T20:47:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-08T20:47:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:47:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-08T20:47:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:47:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:47:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:47:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-12-08T20:47:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:47:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:47:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local loop_instance 2023-12-08T20:47:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:47:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:47:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:47:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:47:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:47:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:47:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-08T20:47:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:47:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:47:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:47:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_33.status 2023-12-08T20:47:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_33.status 2023-12-08T20:47:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.status 2023-12-08T20:47:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_33.status 2023-12-08T20:47:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_33.status 2023-12-08T20:47:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.status 2023-12-08T20:47:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-08T20:47:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-08T20:47:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 183 2023-12-08T20:47:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.status 2023-12-08T20:47:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.status 2023-12-08T20:47:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-08T20:47:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=183 2023-12-08T20:47:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.status 2023-12-08T20:47:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.status 2023-12-08T20:47:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-08T20:47:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 183 = 2023-12-08T20:47:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.status 2023-12-08T20:47:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.status 2023-12-08T20:47:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-08T20:47:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 183 = 2023-12-08T20:47:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:39.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-08T20:47:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.status 2023-12-08T20:47:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.status 2023-12-08T20:47:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-08T20:47:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=183 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 183 = 183 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-08T20:47:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_33': 2023-12-08T20:47:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194ba64d28 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194ba64d28 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:24 2023 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:24 2023 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:24 2023 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: bbf990e6-a3c5-4594-aa42-9c737edf1098 2023-12-08T20:47:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_33 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.export 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.export 2023-12-08T20:47:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.export 2023-12-08T20:47:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.export 2023-12-08T20:47:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.export 2023-12-08T20:47:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.export 2023-12-08T20:47:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_33.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_33.export 2023-12-08T20:47:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_34 2023-12-08T20:47:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-08T20:47:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:47:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + shift 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:47:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:47:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:47:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:47:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:47:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:47:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:47:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:47:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-08T20:47:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + grep -i Replaying 2023-12-08T20:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:echo "state": "Replaying" 2023-12-08T20:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:47:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:47:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:47:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-08T20:47:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_34.status 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_34.status 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.status 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_34.status 2023-12-08T20:47:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_34.status 2023-12-08T20:47:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-08T20:47:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-08T20:47:44.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-12-08T20:47:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 185 2023-12-08T20:47:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.status 2023-12-08T20:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.status 2023-12-08T20:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-08T20:47:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=185 2023-12-08T20:47:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-08T20:47:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 185 = 2023-12-08T20:47:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-08T20:47:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 185 = 2023-12-08T20:47:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-08T20:47:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 185 = 2023-12-08T20:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:47:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.status 2023-12-08T20:47:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-08T20:47:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=185 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 185 = 185 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-08T20:47:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_34': 2023-12-08T20:47:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 11942de507d 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.11942de507d 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:34 2023 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:34 2023 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:34 2023 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 49f707d8-a6ca-42cf-aebb-9130be1a4ef5 2023-12-08T20:47:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_34 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.export 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.export 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.export 2023-12-08T20:47:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.export 2023-12-08T20:47:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.export 2023-12-08T20:47:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.export 2023-12-08T20:47:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_34.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_34.export 2023-12-08T20:47:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_35 2023-12-08T20:47:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-08T20:47:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:47:49.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:47:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state+ 2023-12-08T20:47:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:shift 2023-12-08T20:47:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:47:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:47:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:47:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:47:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:47:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-08T20:47:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:47:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:47:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:47:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:47:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:47:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:47:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:47:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:47:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:47:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:47:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:47:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:47:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:47:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:47:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:47:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-08T20:47:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:47:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:47:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:47:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-08T20:47:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_35.status 2023-12-08T20:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_35.status 2023-12-08T20:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.status 2023-12-08T20:47:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_35.status 2023-12-08T20:47:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_35.status 2023-12-08T20:47:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:47:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-08T20:47:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-08T20:47:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 187 2023-12-08T20:47:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.status 2023-12-08T20:47:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.status 2023-12-08T20:47:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-08T20:47:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:47:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=187 2023-12-08T20:47:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:47:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-08T20:47:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=102 2023-12-08T20:47:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 187 = 102 2023-12-08T20:47:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:47:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-08T20:47:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 187 = 2023-12-08T20:47:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-08T20:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:47:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:47:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:47:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:47:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:47:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 187 = 2023-12-08T20:47:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:47:55.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-12-08T20:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.status 2023-12-08T20:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-08T20:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:47:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:47:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=187 2023-12-08T20:47:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 187 = 187 2023-12-08T20:47:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-08T20:47:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:47:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:47:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:47:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_35': 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194d5af1f1d 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194d5af1f1d 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:47:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:48 2023 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:48 2023 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:48 2023 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 6ff2bf31-0b62-4e55-8ee7-bb0a0e2e75a0 2023-12-08T20:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_35 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.export 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.export 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.export 2023-12-08T20:47:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.export 2023-12-08T20:47:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.export 2023-12-08T20:47:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:47:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.export 2023-12-08T20:47:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_35.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_35.export 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_36 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:47:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:48:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-08T20:48:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:48:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:48:00.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-12-08T20:48:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-08T20:48:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + sleep 0 2023-12-08T20:48:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:48:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:48:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:48:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-08T20:48:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:48:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:48:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:48:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:48:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_36.status 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_36.status 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.status 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_36.status 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_36.status 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-08T20:48:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 189 2023-12-08T20:48:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.status 2023-12-08T20:48:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.status 2023-12-08T20:48:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-08T20:48:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:48:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=189 2023-12-08T20:48:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:48:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-08T20:48:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:02.599 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6856800649567324) > weight (0.25) 2023-12-08T20:48:02.599 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.47104399755774706) > weight (0.25) 2023-12-08T20:48:02.599 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.443600463717459) > weight (0.25) 2023-12-08T20:48:02.599 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6542209725325844) > weight (0.25) 2023-12-08T20:48:02.600 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-08T20:48:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=103 2023-12-08T20:48:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 189 = 103 2023-12-08T20:48:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-08T20:48:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:48:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:48:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:48:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:48:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 189 = 2023-12-08T20:48:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:48:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-08T20:48:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:48:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:48:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:48:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:48:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 189 = 2023-12-08T20:48:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1.6 2023-12-08T20:48:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.status 2023-12-08T20:48:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-08T20:48:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=189 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 189 = 189 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:48:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:48:06.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-12-08T20:48:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-08T20:48:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_36': 2023-12-08T20:48:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:48:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:48:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:48:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194ceabafd8 2023-12-08T20:48:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194ceabafd8 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:40:56 2023 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:40:56 2023 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:40:56 2023 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:48:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 4be21f09-5358-40e3-8098-0e9b567e3808 2023-12-08T20:48:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:48:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-08T20:48:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_36 2023-12-08T20:48:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:48:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.export 2023-12-08T20:48:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.export 2023-12-08T20:48:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.export 2023-12-08T20:48:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.export 2023-12-08T20:48:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:48:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.export 2023-12-08T20:48:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:48:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.export 2023-12-08T20:48:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_36.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_36.export 2023-12-08T20:48:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_37 2023-12-08T20:48:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-08T20:48:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:07.338 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:48:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:48:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:48:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-08T20:48:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:48:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:48:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:48:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-08T20:48:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:48:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:48:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:48:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:48:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:48:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:48:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:48:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-08T20:48:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:48:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:48:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:48:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:48:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:48:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-08T20:48:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:48:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:48:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:48:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:48:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_37.status 2023-12-08T20:48:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_37.status 2023-12-08T20:48:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:48:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.status 2023-12-08T20:48:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_37.status 2023-12-08T20:48:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + echo cluster1-mirror-image_37.statussed 2023-12-08T20:48:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: -e s|[/ ]|_|g 2023-12-08T20:48:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.status 2023-12-08T20:48:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-08T20:48:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-08T20:48:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 191 2023-12-08T20:48:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.status 2023-12-08T20:48:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.status 2023-12-08T20:48:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-08T20:48:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:48:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=191 2023-12-08T20:48:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:48:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.status 2023-12-08T20:48:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.status 2023-12-08T20:48:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-08T20:48:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:48:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:48:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:48:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:48:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 191 = 2023-12-08T20:48:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:48:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.status 2023-12-08T20:48:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.status 2023-12-08T20:48:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-08T20:48:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:48:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:48:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:48:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:48:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 191 = 2023-12-08T20:48:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:48:11.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-12-08T20:48:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.status 2023-12-08T20:48:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.status 2023-12-08T20:48:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-08T20:48:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=191 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 191 = 191 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:48:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:48:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-08T20:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_37': 2023-12-08T20:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194a28d251b 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194a28d251b 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:41:02 2023 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:41:02 2023 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:41:02 2023 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 614d01e3-7add-4c70-abcb-94fcdd953061 2023-12-08T20:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:48:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-08T20:48:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_37 2023-12-08T20:48:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.export 2023-12-08T20:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.export 2023-12-08T20:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.export 2023-12-08T20:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.export 2023-12-08T20:48:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:48:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.export 2023-12-08T20:48:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:48:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.export 2023-12-08T20:48:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_37.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_37.export 2023-12-08T20:48:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_38 2023-12-08T20:48:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-08T20:48:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:48:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-08T20:48:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-08T20:48:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:48:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:48:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:48:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-08T20:48:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:48:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ + local cluster_instance=cluster1 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:grep -i state 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:48:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-08T20:48:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:48:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:48:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:48:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_38.status 2023-12-08T20:48:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_38.status 2023-12-08T20:48:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:48:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.status 2023-12-08T20:48:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_38.status 2023-12-08T20:48:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_38.status 2023-12-08T20:48:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:48:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.status 2023-12-08T20:48:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-08T20:48:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-08T20:48:16.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-12-08T20:48:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:Snapshot ID: 193 2023-12-08T20:48:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.status 2023-12-08T20:48:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.status 2023-12-08T20:48:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-08T20:48:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-08T20:48:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snapshot_id=193 2023-12-08T20:48:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.2 2023-12-08T20:48:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.status 2023-12-08T20:48:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.status 2023-12-08T20:48:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-08T20:48:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:48:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:48:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:48:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:48:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 193 = 2023-12-08T20:48:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.4 2023-12-08T20:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.status 2023-12-08T20:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.status 2023-12-08T20:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-08T20:48:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ true 2023-12-08T20:48:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:-:1.1: Document is empty 2023-12-08T20:48:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: 2023-12-08T20:48:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:^ 2023-12-08T20:48:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id= 2023-12-08T20:48:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 193 = 2023-12-08T20:48:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0.8 2023-12-08T20:48:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.status 2023-12-08T20:48:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.status 2023-12-08T20:48:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-08T20:48:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-08T20:48:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-08T20:48:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local primary_snapshot_id=193 2023-12-08T20:48:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test 193 = 193 2023-12-08T20:48:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-08T20:48:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:48:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:48:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-08T20:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout:rbd image 'image_38': 2023-12-08T20:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: size 128 MiB in 32 objects 2023-12-08T20:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: order 22 (4 MiB objects) 2023-12-08T20:48:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: snapshot_count: 2 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: id: 1194dbe60b5b 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: block_name_prefix: rbd_data.1194dbe60b5b 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: format: 2 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: op_features: 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: flags: 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: create_timestamp: Fri Dec 8 20:41:17 2023 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: access_timestamp: Fri Dec 8 20:41:17 2023 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: modify_timestamp: Fri Dec 8 20:41:17 2023 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: protected: False 2023-12-08T20:48:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring state: enabled 2023-12-08T20:48:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring mode: snapshot 2023-12-08T20:48:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring global id: 2b65f3dc-bcb4-4889-8f5a-69e9969de719 2023-12-08T20:48:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: mirroring primary: false 2023-12-08T20:48:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_38 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local ret=0 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local rmt_export=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.export 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loc_export=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.export 2023-12-08T20:48:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.export 2023-12-08T20:48:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.export 2023-12-08T20:48:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:48:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.export 2023-12-08T20:48:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-08T20:48:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cmp /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.export 2023-12-08T20:48:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rm -f /tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_38.export /tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_38.export 2023-12-08T20:48:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ image=image_39 2023-12-08T20:48:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-08T20:48:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:48:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local snap_name=snap 2023-12-08T20:48:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-08T20:48:22.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-12-08T20:48:22.329 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-08T20:48:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-08T20:48:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:48:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-08T20:48:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:48:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local state=started 2023-12-08T20:48:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local s 2023-12-08T20:48:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 1 2023-12-08T20:48:22.732 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-12-08T20:48:22.733 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-08T20:48:22.733 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-08T20:48:22.733 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-08T20:48:22.734 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f670b1bf700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 31037) UID: 0 2023-12-08T20:48:22.734 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f670b1bf700 -1 osd.1 279 *** Got signal Terminated *** 2023-12-08T20:48:22.734 INFO:tasks.ceph.cluster2.osd.1.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f670b1bf700 -1 osd.1 279 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-08T20:48:22.734 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f588456c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 31040) UID: 0 2023-12-08T20:48:22.734 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f588456c700 -1 osd.2 279 *** Got signal Terminated *** 2023-12-08T20:48:22.734 INFO:tasks.ceph.cluster2.osd.2.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f588456c700 -1 osd.2 279 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-08T20:48:22.735 INFO:tasks.ceph.cluster2.mon.a.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f18d798d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 30883) UID: 0 2023-12-08T20:48:22.735 INFO:tasks.ceph.cluster2.mon.a.smithi158.stderr:2023-12-08T20:48:22.723+0000 7f18d798d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-08T20:48:22.735 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:48:22.722+0000 7fb52d4be700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 34012) UID: 0 2023-12-08T20:48:22.735 INFO:tasks.ceph.cluster2.mgr.x.smithi069.stderr:2023-12-08T20:48:22.722+0000 7fb52d4be700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-08T20:48:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-08T20:48:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:48:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local test_state=started 2023-12-08T20:48:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_result 2023-12-08T20:48:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local current_state=stopped 2023-12-08T20:48:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemons+ grep -i state 2023-12-08T20:48:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr: cluster1 rbd mirror status mirror/image_39 2023-12-08T20:48:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_instance=cluster1 2023-12-08T20:48:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance=cluster1 2023-12-08T20:48:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local loop_instance 2023-12-08T20:48:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sleep 0 2023-12-08T20:48:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ cluster1 != cluster1 ] 2023-12-08T20:48:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ seq 0 3 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1:0 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ shift 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=cluster 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval cluster=cluster1 2023-12-08T20:48:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-08T20:48:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1:0 2023-12-08T20:48:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster1 2023-12-08T20:48:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance 2023-12-08T20:48:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local val=cluster1:0 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster_var_name=local_cluster 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local instance_var_name=instance 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ cluster=cluster1 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ 0 = cluster1:0 ] 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval local_cluster=cluster1 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local_cluster=cluster1 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ eval instance=0 2023-12-08T20:48:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ instance=0 2023-12-08T20:48:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-08T20:48:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-08T20:48:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-08T20:48:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ status_result= "state": "Replaying" 2023-12-08T20:48:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo "state": "Replaying" 2023-12-08T20:48:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ grep -i Replaying 2023-12-08T20:48:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stdout: "state": "Replaying" 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ current_state=started 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ test started = started 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ return 0 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = journal ] 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ [ snapshot = snapshot ] 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_cluster=cluster1 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster2-mirror-image_39.status 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster2-mirror-image_39.status 2023-12-08T20:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:48:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local status_log=/tmp/tmp.b8SaxlVk4h/cluster2-mirror-image_39.status 2023-12-08T20:48:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mkfname cluster1-mirror-image_39.status 2023-12-08T20:48:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ echo cluster1-mirror-image_39.status 2023-12-08T20:48:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ sed -e s|[/ ]|_|g 2023-12-08T20:48:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local local_status_log=/tmp/tmp.b8SaxlVk4h/cluster1-mirror-image_39.status 2023-12-08T20:48:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-08T20:48:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local cluster=cluster2 2023-12-08T20:48:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local pool=mirror 2023-12-08T20:48:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ local image=image_39 2023-12-08T20:48:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi158.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-08T20:48:26.396 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.750705468972632) > weight (0.25) 2023-12-08T20:48:26.396 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4482636380216706) > weight (0.25) 2023-12-08T20:48:26.396 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-08T20:48:26.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-08T20:48:26.397 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-08T20:48:26.397 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-08T20:48:26.397 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:48:26.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi158.stderr:2023-12-08T20:48:26.387+0000 7f0c81f40700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46820) UID: 1000 2023-12-08T20:48:26.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi158.stderr:daemon-helper: command failed with exit status 1 2023-12-08T20:48:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T20:48:32.499 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-08T20:48:32.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-08T20:48:32.499 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-08T20:48:32.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-08T20:48:32.502 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-08T20:49:55.601 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9956898258071671) > weight (0.25) 2023-12-08T20:49:55.601 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-08T20:49:55.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-08T20:49:55.602 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-08T20:49:55.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:49:55.594+0000 7f5991e3f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 45102) UID: 1000 2023-12-08T20:50:02.603 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-08T20:50:02.604 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:50:17.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.252862694305687) > weight (0.25) 2023-12-08T20:50:17.504 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-08T20:50:17.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-08T20:50:17.505 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-08T20:50:17.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T20:50:17.498+0000 7fe0a165c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 45519) UID: 1000 2023-12-08T20:50:24.505 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-08T20:50:24.505 INFO:teuthology.orchestra.run:waiting for 600 2023-12-08T20:50:24.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.990+0000 7f5990e3d700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:24.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:24.990+0000 7ffa87672700 -1 rbd::mirror::LeaderWatcher: 0x55c7d06dc300 handle_break_leader_lock: error breaking leader lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:24.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:24.990+0000 7fbdf18e0700 -1 rbd::mirror::LeaderWatcher: 0x55902508c300 handle_break_leader_lock: error breaking leader lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:50:24.990+0000 7f5ecfde9700 -1 rbd::mirror::LeaderWatcher: 0x55cd9c132300 handle_break_leader_lock: error breaking leader lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.990+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_remove_instance_object: error removing rbd_mirror_instance.4921 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.990+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.994+0000 7f5990e3d700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.998+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_remove_instance_object: error removing rbd_mirror_instance.4503 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.998+0000 7f599163e700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.998+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.998+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_remove_instance_object: error removing rbd_mirror_instance.5148 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:25.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T20:50:24.998+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:30.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:50:30.930+0000 7f5eb15ac700 -1 rbd::mirror::PoolReplayer: 0x55cd9a882f00 list_mirroring_namespaces: failed to list namespaces: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:31.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:50:31.038+0000 7f5ed3634680 -1 rbd::mirror::Mirror: 0x55cd9842b8c0 update_pool_replayers: restarting blocklisted pool replayer for uuid: 11d951e7-3456-4114-ba07-859714a29881 cluster: cluster2 client: client.mirror 2023-12-08T20:50:31.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:50:31.038+0000 7f5ebb5c0700 -1 rbd::mirror::InstanceWatcher: 0x55cd99163880 handle_remove_instance_object: error removing rbd_mirror_instance.5148 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:31.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi158.stderr:2023-12-08T20:50:31.038+0000 7f5ebbdc1700 -1 rbd::mirror::InstanceWatcher: 0x55cd99163880 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:34.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:34.990+0000 7ffa87672700 -1 rbd::mirror::LeaderWatcher: 0x55c7d06dc300 handle_acquire_leader_lock: error acquiring lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:35.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:34.990+0000 7fbdf10df700 -1 rbd::mirror::LeaderWatcher: 0x55902508c300 handle_acquire_leader_lock: error acquiring lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:35.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:34.990+0000 7ffa86e71700 -1 rbd::mirror::LeaderWatcher: 0x55c7d06dc300 handle_get_locker: error retrieving leader locker: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:35.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:34.990+0000 7fbdf18e0700 -1 rbd::mirror::LeaderWatcher: 0x55902508c300 handle_get_locker: error retrieving leader locker: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:40.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:39.990+0000 7ffa86e71700 -1 rbd::mirror::LeaderWatcher: 0x55c7d06dc300 handle_get_locker: error retrieving leader locker: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:40.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:39.990+0000 7fbdf10df700 -1 rbd::mirror::LeaderWatcher: 0x55902508c300 handle_get_locker: error retrieving leader locker: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:42.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:42.802+0000 7ffa68e35700 -1 rbd::mirror::PoolReplayer: 0x55c7cee24f00 list_mirroring_namespaces: failed to list namespaces: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:43.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:43.502+0000 7ffa8aebd680 -1 rbd::mirror::Mirror: 0x55c7cc9e18c0 update_pool_replayers: restarting blocklisted pool replayer for uuid: 11d951e7-3456-4114-ba07-859714a29881 cluster: cluster2 client: client.mirror 2023-12-08T20:50:43.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:43.506+0000 7ffa7364a700 -1 rbd::mirror::InstanceWatcher: 0x55c7cd71f500 handle_remove_instance_object: error removing rbd_mirror_instance.4503 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:43.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi158.stderr:2023-12-08T20:50:43.506+0000 7ffa72e49700 -1 rbd::mirror::InstanceWatcher: 0x55c7cd71f500 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:45.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:44.994+0000 7fbdf18e0700 -1 rbd::mirror::LeaderWatcher: 0x55902508c300 handle_get_locker: error retrieving leader locker: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:45.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:45.850+0000 7fbdd30a3700 -1 rbd::mirror::PoolReplayer: 0x5590237d0f00 list_mirroring_namespaces: failed to list namespaces: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:46.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:46.010+0000 7fbdf512b680 -1 rbd::mirror::Mirror: 0x5590213878c0 update_pool_replayers: restarting blocklisted pool replayer for uuid: 11d951e7-3456-4114-ba07-859714a29881 cluster: cluster2 client: client.mirror 2023-12-08T20:50:46.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:46.010+0000 7fbddd8b8700 -1 rbd::mirror::InstanceWatcher: 0x5590220c1500 handle_remove_instance_object: error removing rbd_mirror_instance.4921 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T20:50:46.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi158.stderr:2023-12-08T20:50:46.010+0000 7fbddd0b7700 -1 rbd::mirror::InstanceWatcher: 0x5590220c1500 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.743 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.1 2023-12-08T21:00:06.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 6 2023-12-08T21:00:06.744 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-08T21:00:06.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:*** Caught signal (Aborted) ** 2023-12-08T21:00:06.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: in thread 7f5994e89680 thread_name:rbd-mirror 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5995713420] 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f59955e7e30] 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 4: (rbd::mirror::LeaderWatcher::shut_down()+0x10b) [0x5578521b3e4b] 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x94) [0x557852167de4] 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x557852168300] 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x44) [0x55785215d264] 2023-12-08T21:00:06.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x2c) [0x55785215d24c] 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5578521570b5] 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 10: main() 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 11: __libc_start_main() 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 12: _start() 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:2023-12-08T21:00:06.735+0000 7f5994e89680 -1 *** Caught signal (Aborted) ** 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: in thread 7f5994e89680 thread_name:rbd-mirror 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5995713420] 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f59955e7e30] 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 4: (rbd::mirror::LeaderWatcher::shut_down()+0x10b) [0x5578521b3e4b] 2023-12-08T21:00:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x94) [0x557852167de4] 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x557852168300] 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x44) [0x55785215d264] 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x2c) [0x55785215d24c] 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5578521570b5] 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 10: main() 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 11: __libc_start_main() 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 12: _start() 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T21:00:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:06.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2414> 2023-12-08T20:49:55.594+0000 7f5991e3f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 45102) UID: 1000 2023-12-08T21:00:06.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2173> 2023-12-08T20:50:24.990+0000 7f5990e3d700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2170> 2023-12-08T20:50:24.990+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_remove_instance_object: error removing rbd_mirror_instance.4921 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2165> 2023-12-08T20:50:24.990+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2160> 2023-12-08T20:50:24.994+0000 7f5990e3d700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2157> 2023-12-08T20:50:24.998+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_remove_instance_object: error removing rbd_mirror_instance.4503 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2154> 2023-12-08T20:50:24.998+0000 7f599163e700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2151> 2023-12-08T20:50:24.998+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2146> 2023-12-08T20:50:24.998+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_remove_instance_object: error removing rbd_mirror_instance.5148 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -2143> 2023-12-08T20:50:24.998+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 0> 2023-12-08T21:00:06.735+0000 7f5994e89680 -1 *** Caught signal (Aborted) ** 2023-12-08T21:00:06.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: in thread 7f5994e89680 thread_name:rbd-mirror 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5995713420] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f59955e7e30] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 4: (rbd::mirror::LeaderWatcher::shut_down()+0x10b) [0x5578521b3e4b] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x94) [0x557852167de4] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x557852168300] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x44) [0x55785215d264] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x2c) [0x55785215d24c] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5578521570b5] 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 10: main() 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 11: __libc_start_main() 2023-12-08T21:00:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 12: _start() 2023-12-08T21:00:06.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T21:00:06.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:06.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8170> 2023-12-08T20:49:55.594+0000 7f5991e3f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 45102) UID: 1000 2023-12-08T21:00:06.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8169> 2023-12-08T20:50:24.990+0000 7f5990e3d700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8168> 2023-12-08T20:50:24.990+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_remove_instance_object: error removing rbd_mirror_instance.4921 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8167> 2023-12-08T20:50:24.990+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28388 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8166> 2023-12-08T20:50:24.994+0000 7f5990e3d700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8165> 2023-12-08T20:50:24.998+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_remove_instance_object: error removing rbd_mirror_instance.4503 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8164> 2023-12-08T20:50:24.998+0000 7f599163e700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8163> 2023-12-08T20:50:24.998+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28008 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8162> 2023-12-08T20:50:24.998+0000 7f597d616700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_remove_instance_object: error removing rbd_mirror_instance.5148 object: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8161> 2023-12-08T20:50:24.998+0000 7f597ce15700 -1 rbd::mirror::InstanceWatcher: 0x557856f28708 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: -8160> 2023-12-08T21:00:06.735+0000 7f5994e89680 -1 *** Caught signal (Aborted) ** 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: in thread 7f5994e89680 thread_name:rbd-mirror 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5995713420] 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f59955e7e30] 2023-12-08T21:00:06.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 4: (rbd::mirror::LeaderWatcher::shut_down()+0x10b) [0x5578521b3e4b] 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0x94) [0x557852167de4] 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x557852168300] 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x44) [0x55785215d264] 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x2c) [0x55785215d24c] 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5578521570b5] 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 10: main() 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 11: __libc_start_main() 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 12: _start() 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T21:00:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:07.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi158.stderr:daemon-helper: command crashed with signal 6 2023-12-08T21:00:07.944 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2023-12-08T21:00:07.944 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-08T21:00:08.449 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2023-12-08T21:00:08.450 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2023-12-08T21:00:08.450 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2023-12-08T21:00:08.450 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2023-12-08T21:00:08.451 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2023-12-08T21:00:08.451 INFO:tasks.ceph.cluster1.osd.0.smithi069.stderr:2023-12-08T21:00:08.442+0000 7feade5ef700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 30914) UID: 0 2023-12-08T21:00:08.451 INFO:tasks.ceph.cluster1.osd.0.smithi069.stderr:2023-12-08T21:00:08.442+0000 7feade5ef700 -1 osd.0 284 *** Got signal Terminated *** 2023-12-08T21:00:08.451 INFO:tasks.ceph.cluster1.osd.0.smithi069.stderr:2023-12-08T21:00:08.442+0000 7feade5ef700 -1 osd.0 284 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-08T21:00:08.452 INFO:tasks.ceph.cluster1.osd.1.smithi069.stderr:2023-12-08T21:00:08.442+0000 7f819fdd5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 30912) UID: 0 2023-12-08T21:00:08.452 INFO:tasks.ceph.cluster1.osd.1.smithi069.stderr:2023-12-08T21:00:08.442+0000 7f819fdd5700 -1 osd.1 284 *** Got signal Terminated *** 2023-12-08T21:00:08.452 INFO:tasks.ceph.cluster1.osd.1.smithi069.stderr:2023-12-08T21:00:08.442+0000 7f819fdd5700 -1 osd.1 284 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-08T21:00:08.452 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T21:00:08.442+0000 7f2c7a9c9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 30915) UID: 0 2023-12-08T21:00:08.452 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T21:00:08.442+0000 7f2c7a9c9700 -1 osd.2 284 *** Got signal Terminated *** 2023-12-08T21:00:08.452 INFO:tasks.ceph.cluster1.osd.2.smithi069.stderr:2023-12-08T21:00:08.442+0000 7f2c7a9c9700 -1 osd.2 284 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-08T21:00:08.453 INFO:tasks.ceph.cluster1.mon.a.smithi069.stderr:2023-12-08T21:00:08.442+0000 7ff256aa5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 30730) UID: 0 2023-12-08T21:00:08.453 INFO:tasks.ceph.cluster1.mon.a.smithi069.stderr:2023-12-08T21:00:08.442+0000 7ff256aa5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-08T21:00:08.453 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T21:00:08.442+0000 7ff190fe0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 30732) UID: 0 2023-12-08T21:00:08.453 INFO:tasks.ceph.cluster1.mgr.x.smithi069.stderr:2023-12-08T21:00:08.442+0000 7ff190fe0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-08T21:00:28.658 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.1 2023-12-08T21:00:28.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 6 2023-12-08T21:00:28.659 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-08T21:00:28.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:*** Caught signal (Aborted) ** 2023-12-08T21:00:28.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: in thread 7fe0a46a6680 thread_name:rbd-mirror 2023-12-08T21:00:28.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:28.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fe0a4f30420] 2023-12-08T21:00:28.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:28.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fe0a4e04e30] 2023-12-08T21:00:28.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 4: /lib/librados.so.2(+0xc20db) [0x7fe0ae2f60db] 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55cace243888] 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::ClusterWatcher::refresh_pools()+0x74) [0x55cace246284] 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 7: (rbd::mirror::Mirror::run()+0xa6) [0x55cace207a16] 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 8: main() 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 9: __libc_start_main() 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 10: _start() 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:2023-12-08T21:00:28.651+0000 7fe0a46a6680 -1 *** Caught signal (Aborted) ** 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: in thread 7fe0a46a6680 thread_name:rbd-mirror 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fe0a4f30420] 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fe0a4e04e30] 2023-12-08T21:00:28.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 4: /lib/librados.so.2(+0xc20db) [0x7fe0ae2f60db] 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55cace243888] 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::ClusterWatcher::refresh_pools()+0x74) [0x55cace246284] 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 7: (rbd::mirror::Mirror::run()+0xa6) [0x55cace207a16] 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 8: main() 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 9: __libc_start_main() 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 10: _start() 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T21:00:28.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:28.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: -1670> 2023-12-08T20:50:17.498+0000 7fe0a165c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 45519) UID: 1000 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 0> 2023-12-08T21:00:28.651+0000 7fe0a46a6680 -1 *** Caught signal (Aborted) ** 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: in thread 7fe0a46a6680 thread_name:rbd-mirror 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fe0a4f30420] 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fe0a4e04e30] 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 4: /lib/librados.so.2(+0xc20db) [0x7fe0ae2f60db] 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55cace243888] 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::ClusterWatcher::refresh_pools()+0x74) [0x55cace246284] 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 7: (rbd::mirror::Mirror::run()+0xa6) [0x55cace207a16] 2023-12-08T21:00:28.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 8: main() 2023-12-08T21:00:28.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 9: __libc_start_main() 2023-12-08T21:00:28.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 10: _start() 2023-12-08T21:00:28.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T21:00:28.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:28.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: -6714> 2023-12-08T20:50:17.498+0000 7fe0a165c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 45519) UID: 1000 2023-12-08T21:00:28.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: -6713> 2023-12-08T21:00:28.651+0000 7fe0a46a6680 -1 *** Caught signal (Aborted) ** 2023-12-08T21:00:28.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: in thread 7fe0a46a6680 thread_name:rbd-mirror 2023-12-08T21:00:28.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:28.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: ceph version 16.2.14-365-g451ceae1 (451ceae11b5d4caee05e94372fe658621eefc0b1) pacific (stable) 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fe0a4f30420] 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2: pthread_cond_wait() 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fe0a4e04e30] 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 4: /lib/librados.so.2(+0xc20db) [0x7fe0ae2f60db] 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 5: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55cace243888] 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 6: (rbd::mirror::ClusterWatcher::refresh_pools()+0x74) [0x55cace246284] 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 7: (rbd::mirror::Mirror::run()+0xa6) [0x55cace207a16] 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 8: main() 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 9: __libc_start_main() 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 10: _start() 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-08T21:00:28.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr: 2023-12-08T21:00:29.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi158.stderr:daemon-helper: command crashed with signal 6 2023-12-08T21:24:51.968 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-12-08T21:24:51.985 DEBUG:teuthology.task.console_log:Killing console logger for smithi069 2023-12-08T21:24:51.986 DEBUG:teuthology.task.console_log:Killing console logger for smithi158 2023-12-08T21:24:51.986 DEBUG:teuthology.exit:Finished running handlers